Magnetic tracking of glove fingertips with peripheral devices

ABSTRACT

A method is provided, including: activating a plurality of glove emitters positioned on a glove interface object; using a plurality of proximity sensors positioned at fingertip portions of the glove interface object to determine a proximity of the fingertip portions to the glove emitters; in response to determining a location of the glove interface object within a predefined distance of a peripheral device, activating a plurality of peripheral emitters positioned at the peripheral device, and transitioning, from using the proximity sensors to determine the proximity of the fingertip portions to the glove emitters, to using the proximity sensors to determine a proximity of the fingertip portions to the peripheral emitters.

CLAIM OF PRIORITY

This application claims priority to U.S. Provisional Application No.62/118,734, filed Feb. 20, 2015, entitled “Magnetic Tracking of GloveFingertips,” the disclosure of which is incorporated by referenceherein.

RELATED APPLICATIONS

This application is related to U.S. application Ser. No. ______(Attorney Docket SONYP226A), filed the same day as the presentapplication, entitled “Magnetic Tracking of Glove Fingertips,” and toU.S. application Ser. No. 14/517,741, filed Oct. 17, 2014, entitled“Glove Interface Object,” and to U.S. application Ser. No. 14/517,733,filed Oct. 17, 2014, entitled “Thumb Controller,” the disclosures ofwhich are incorporated by reference herein.

BACKGROUND

1. Field of the Invention

The present invention relates to magnetic tracking of glove fingertipsand associated methods and systems.

2. Description of the Related Art

The video game industry has seen many changes over the years. Ascomputing power has expanded, developers of video games have likewisecreated game software that takes advantage of these increases incomputing power. To this end, video game developers have been codinggames that incorporate sophisticated operations and mathematics toproduce a very realistic game experience.

Example gaming platforms, may be the Sony Playstation®, SonyPlaystation2® (PS2), Sony Playstation3® (PS3), and Sony Playstation4®(PS4), each of which is sold in the form of a game console. As is wellknown, the game console is designed to connect to a monitor (usually atelevision) and enable user interaction through handheld controllers.The game console is designed with specialized processing hardware,including a CPU, a graphics synthesizer for processing intensivegraphics operations, a vector unit for performing geometrytransformations, and other glue hardware, firmware, and software. Thegame console is further designed with an optical disc tray for receivinggame compact discs for local play through the game console. Onlinegaming is also possible, where a user can interactively play against orwith other users over the Internet. As game complexity continues tointrigue players, game and hardware manufacturers have continued toinnovate to enable additional interactivity and computer programs.

A growing trend in the computer gaming industry is to develop games thatincrease the interaction between the user and the gaming system. One wayof accomplishing a richer interactive experience is to use wireless gamecontrollers whose movement is tracked by the gaming system in order totrack the player's movements and use these movements as inputs for thegame. Generally speaking, gesture input refers to having an electronicdevice such as a computing system, video game console, smart appliance,etc., react to some gesture made by the player and captured by theelectronic device.

Another way of accomplishing a more immersive interactive experience isto use a head-mounted display. A head-mounted display is worn by theuser and can be configured to present various graphics, such as a viewof a virtual space. The graphics presented on a head-mounted display cancover a large portion or even all of a user's field of view. Hence, ahead-mounted display can provide a visually immersive experience to theuser.

Another growing trend in the industry involves the development ofcloud-based gaming systems. Such systems may include a remote processingserver that executes a game application, and communicates with a localthin client that can be configured to receive input from users andrender video on a display.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments of the present invention provide for magnetic tracking ofglove fingertips and associated methods and systems.

Broadly speaking a magnetic tracking system to track fingertips andknuckles is provided to create an input device that captures hand/fingerpose. In one embodiment, a number of magnetic sensors (e.g. Hall effectsensors) are placed on the hand (e.g. one per fingertip), and threespatially separated electromagnets are placed on the wrists. Theelectromagnets can be cycled in turn to allow the sensors to makeseparate readings from each one, and then the three readings per sensorcan be used to compute a three-dimensional (3D) position of thefingertip relative to the wrist. Given the 3D position of the fingertip,then it is possible to use inverse kinematics to reduce the pose of thefinger (e.g. identify which knuckles are bent and the amount of bending)to one possible configuration (or several almost identicalconfigurations). Additionally, magnetic emitters can be placed in otherdevices, such as a physical controller device, and may be used with thefingertip sensors to determine their pose relative to these devices, toshow proper interplay between the hand and the device (e.g. a user canbe provided with a display rendering that shows their hand grasping acontroller device).

In one embodiment, a glove interface object is provided, comprising: aplurality of electromagnets positioned at a wrist area of the gloveinterface object; a plurality of magnetic sensors respectivelypositioned at fingertip areas of the glove interface object, whereineach magnetic sensor is configured to generate data indicating distancesto each of the electromagnets when each of the electromagnets isactivated; a controller configured to control activation of theelectromagnets and reading of the magnetic sensors in a time-divisionmultiplexed arrangement, wherein each of the magnetic sensors is readduring activation of a single electromagnet; a transmitter configured totransmit data derived from the reading of the magnetic sensors to acomputing device for processing to generate data representing a pose ofa virtual hand, the virtual hand capable of being rendered in a virtualenvironment presented on a head-mounted display.

In one embodiment, the time-division multiplexed arrangement is definedby a repeated pattern of activation of the electromagnets that providesfor activation of each of the electromagnets during separate timeperiods, and reading of each of the magnetic sensors during the timeperiod of activation of a given one of the electromagnets.

In one embodiment, the plurality of electromagnets defines at leastthree electromagnets that are positioned on the wrist area in anon-collinear arrangement.

In one embodiment, the plurality of magnetic sensors defines fivemagnetic sensors respectively positioned at five fingertip areas of theglove interface object.

In one embodiment, each of the plurality of magnetic sensors isconfigured to generate a voltage in response to a magnetic fieldgenerated by one of the electromagnets.

In one embodiment, the glove interface object further comprises: anilluminated trackable object that is configured to be tracked based onanalysis of captured images of the glove interface object in aninteractive environment, the tracking of the illuminated trackableobject defining a location of the virtual hand in a virtual environment.

In one embodiment, the glove interface object further comprises: atleast one inertial sensor selected from the group consisting of anaccelerometer, a gyroscope, and a magnetometer.

In another embodiment, a method is provided, comprising: seriallyactivating and deactivating a plurality of electromagnets that arepositioned at a wrist portion of a glove interface object, so as todefine periods of activation for each of the electromagnets that aresubstantially non-overlapping; during the activation of each one of theelectromagnets, using a plurality of magnetic sensors to sense astrength of a magnetic field generated by the electromagnet that isactivated, the plurality of magnetic sensors being respectivelypositioned at fingertip portions of the glove interface object;processing the sensed strengths of the magnetic fields to generate dataderived from the sensed strengths of the magnetic fields; sending thedata derived from the sensed strengths of the magnetic fields to acomputing device for processing generate data representing a pose of avirtual hand, the virtual hand capable of being rendered in a virtualenvironment presented on a head-mounted display, such that the pose ofthe virtual hand is substantially similar to a physical pose of theglove interface object.

In one embodiment, processing the sensed strengths of the magneticfields includes determining distances from each of the magnetic sensorsto each of the electromagnets, and determining a relative location ofeach magnetic sensor to the plurality of electromagnets based on thedetermined distances.

In one embodiment, the processing to define the pose of the virtual handincludes processing the relative location of each magnetic sensor todefine a pose for a corresponding virtual finger on the virtual hand.

In one embodiment, the plurality of electromagnets includes at leastthree electromagnets positioned on the wrist portion of the gloveinterface object in a non-collinear arrangement.

In one embodiment, the relative location of a given magnetic sensor tothe plurality of electromagnets is defined by an intersection of radii,the radii having origins defined by each of the electromagnets andmagnitudes defined by the determined distances from the given magneticsensor to each of the electromagnets.

In one embodiment, serially activating and deactivating the plurality ofelectromagnets defines a repetitive cycle of the periods of activationof the electromagnets.

In one embodiment, each of the plurality of magnetic sensors isconfigured to generate a voltage in response to a magnetic fieldgenerated by one of the electromagnets.

In another embodiment, a method is provided, comprising: activating afirst electromagnet positioned on a wrist portion of a glove interfaceobject, the activation of the first electromagnet producing a firstmagnetic field; measuring a strength of the first magnetic field at eachof a plurality of fingertip portions of the glove interface object;deactivating the first electromagnet; activating a second electromagnetpositioned on the wrist portion of the glove interface object, theactivation of the second electromagnet producing a second magneticfield; measuring a strength of the second magnetic field at each of theplurality of fingertip portions of the glove interface object;deactivating the second electromagnet; activating a third electromagnetpositioned on a wrist portion of the glove interface object, theactivation of the third electromagnet producing a third magnetic field;measuring a strength of the third magnetic field at each of theplurality of fingertip portions of the glove interface object;deactivating the third electromagnet; for each of the fingertip portionsof the glove interface object, generating location data that indicates alocation of the fingertip portion based on the measured strength of thefirst, second, and third magnetic fields at the fingertip portion;sending the location data to a computing device for processing togenerate data representing a configuration of a virtual hand, thevirtual hand capable of being rendered in a virtual environmentpresented on a head-mounted display, such that the configuration of thevirtual hand is substantially similar to a physical configuration of theglove interface object.

In one embodiment, the activating and deactivating of the firstelectromagnet defines a period of activation for the first electromagnetduring which the measuring of the strength of the first magnetic fieldis performed; wherein the activating and deactivating of the secondelectromagnet defines a period of activation for the secondelectromagnet during which the measuring of the strength of the secondmagnetic field is performed; wherein the activating and deactivating ofthe third electromagnet defines a period of activation for the thirdelectromagnet during which the measuring of the strength of the thirdmagnetic field is performed; wherein the periods of activation for thefirst, second, and third electromagnets are substantiallynon-overlapping.

In one embodiment, the method further comprises: cyclically performingeach of the operations of the method, so as to provide real-timecorrespondence between the configuration of the virtual hand and thephysical configuration of the glove interface object.

In one embodiment, the location of a given fingertip portion is definedby an intersection of radii, the radii having origins defined by each ofthe electromagnets and magnitudes defined by distances from the givenfingertip portion to each of the electromagnets that are determined fromthe measured strengths of the magnetic fields.

In one embodiment, the first, second, and third electromagnets arepositioned on the wrist portion of the glove interface object in anon-collinear arrangement.

In one embodiment, measuring the strength of the magnetic fields at eachof the plurality of fingertip portions is performed by Hall effectsensors positioned at the plurality of fingertip portions.

In another embodiment, a method is provided, comprising: using an imagecapture device to capture images of an interactive environment;processing the captured images to track a location of a trackable objecton a glove interface object in the interactive environment, the trackedlocation of the trackable object defining a location of a virtual handin a virtual environment; receiving finger proximity data from aplurality of proximity sensors positioned at fingertip portions of theglove interface object, the finger proximity data indicating distancesto each of a plurality of emitters positioned on the glove interfaceobject; processing the proximity data to identify locations of thefingertip portions relative to the emitters; applying the locations ofthe fingertip portions to define poses of virtual fingers of the virtualhand in the virtual environment for rendering on a head-mounted display.

In one embodiment, receiving finger proximity data is defined fromactivation of the emitters and reading of the proximity sensors in atime-division multiplexed arrangement, wherein each of the proximitysensors is read during activation of a single emitter, wherein thetime-division multiplexed arrangement is defined by a repeated patternof activation of the emitters that provides for activation of each ofthe emitters during separate time periods, and reading of each of theproximity sensors during the time period of activation of a given one ofthe emitters.

In one embodiment, the plurality of emitters defines at least threeelectromagnets that are positioned on a wrist portion of the gloveinterface object in a non-collinear arrangement.

In one embodiment, the plurality of proximity sensors defines fivemagnetic sensors respectively positioned at five fingertip portions ofthe glove interface object.

In one embodiment, the trackable object is illuminated to facilitateidentification in the captured images.

In one embodiment, the method further comprises: receiving orientationdata from an orientation sensor defined on the glove interface object;processing the orientation data to define an orientation of the virtualhand in the virtual environment.

In one embodiment, the orientation sensor is selected from the groupconsisting of an accelerometer, a gyroscope, and a magnetometer.

In another embodiment, a method is provided, comprising: activating aplurality of glove emitters positioned on a glove interface object;using a plurality of proximity sensors positioned at fingertip portionsof the glove interface object to determine a proximity of the fingertipportions to the glove emitters; in response to determining a location ofthe glove interface object within a predefined distance of a peripheraldevice, activating a plurality of peripheral emitters positioned at theperipheral device, and transitioning, from using the proximity sensorsto determine the proximity of the fingertip portions to the gloveemitters, to using the proximity sensors to determine a proximity of thefingertip portions to the peripheral emitters.

In one embodiment, transitioning includes terminating the activation ofthe glove emitters.

In one embodiment, activating the glove emitters defines a cyclicalactivation sequence of the glove emitters, wherein during a singleactivation time period for a given glove emitter, proximity data is readfrom each of the proximity sensors.

In one embodiment, determining the location of the glove interfaceobject within the predefined distance of the peripheral device includesprocessing captured image data of an interactive environment to identifythe location of the glove interface object and a location of theperipheral device.

In one embodiment, determining the location of the glove interfaceobject includes activating a secondary peripheral emitter on theperipheral device, and using the proximity sensors to determine aproximity of the fingertip portions to the secondary peripheral emitter;and, wherein transitioning includes terminating the activation of thesecondary peripheral emitter.

In one embodiment, the glove emitters are positioned at a wrist portionof the glove interface object.

In one embodiment, the glove emitters and the peripheral emitters aredefined by electromagnets, and the proximity sensors are defined bymagnetic sensors.

In another embodiment, a system for interfacing with an interactiveapplication is provided, comprising: a glove interface object, the gloveinterface object including, a plurality of glove emitters, a pluralityof proximity sensors positioned at fingertip portions of the gloveinterface object, the proximity sensors being configured to indicate aproximity of the fingertip portions to the glove emitters, a glovecontroller configured to control activation of the plurality of gloveemitters and reading of the proximity sensors; and, a peripheral device,the peripheral device including, a plurality of peripheral emitters, aperipheral controller configured to activate the peripheral emitters inresponse to a location of the glove interface object being determinedwithin a predefined distance of the peripheral device; wherein theproximity sensors, further in response to the location of the gloveinterface object being determined within the predefined distance,transition, from indicating the proximity of the fingertip portions tothe glove emitters, to indicating a proximity of the fingertip portionsto the peripheral emitters.

In one embodiment, the glove controller is configured to terminate theactivation of the glove emitters in response to the location of theglove interface object being determined within the predefined distance.

In one embodiment, the glove controller is configured to define acyclical activation sequence of the glove emitters, wherein during asingle activation time period for a given glove emitter, proximity datais read from each of the proximity sensors.

In one embodiment, the peripheral device further includes, a secondaryperipheral emitter, wherein the peripheral controller is configured tocontrol the activation of the secondary peripheral emitter, theperipheral emitter being used for determining the location of the gloveinterface object; wherein the glove controller is configured to read theproximity sensors to determine a proximity of the fingertip portions tothe secondary peripheral emitter that defines the location of the gloveinterface object; and wherein the peripheral controller is configured,further in response to the location of the glove interface object beingdetermined within the predefined distance, to terminate activation ofthe secondary peripheral emitter.

In one embodiment, the glove emitters are positioned at a wrist portionof the glove interface object.

In one embodiment, the glove emitters and the peripheral emitters aredefined by electromagnets, and the proximity sensors are defined bymagnetic sensors.

Other aspects of the invention will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, illustrating by way of example the principles of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 illustrates a system for interactive gameplay of a video game, inaccordance with an embodiment of the invention.

FIG. 2 illustrates a head-mounted display (HMD), in accordance with anembodiment of the invention.

FIG. 3 conceptually illustrates the function of a HMD in conjunctionwith an executing video game, in accordance with an embodiment of theinvention.

FIG. 4A illustrates a glove interface object having a plurality ofemitters and proximity sensors defined thereon, in accordance with anembodiment of the invention.

FIG. 4B illustrates a glove interface object having multiple proximitysensors defined thereon, in accordance with an embodiment of theinvention.

FIG. 5 is a diagram conceptually illustrating the operation of severalemitters and sensors of a glove interface object in a time divisionmultiplexed arrangement, in accordance with an embodiment of theinvention.

FIG. 6 illustrates several graphs showing the application of power tovarious electromagnets of a glove interface object, in accordance withan embodiment of the invention.

FIG. 7 illustrates a glove interface object configured for use with aperipheral device, in accordance with an embodiment of the invention.

FIG. 8 illustrates a pair of glove interface objects configured tointeract with a keyboard peripheral device, in accordance with anembodiment of the invention.

FIG. 9 illustrates a peripheral device having long-range and close rangeemitters configured to provide for course tracking and fine tracking ofone or more glove interface objects having proximity sensors, inaccordance with an embodiment of the invention.

FIGS. 10A and 10B schematically illustrate a system for interfacing withan interactive application using a glove interface object, in accordancewith an embodiment of the invention.

FIG. 11 illustrates components of a glove interface object, inaccordance with an embodiment of the invention.

FIG. 12 illustrates components of a head-mounted display, in accordancewith an embodiment of the invention.

FIG. 13 is a block diagram of a Game System, according to variousembodiments of the invention.

DETAILED DESCRIPTION

The following embodiments provide a glove interface object andassociated systems, methods, and apparatuses.

In one embodiment, the methods, systems, image capture objects, sensorsand associated interfaces objects (e.g., gloves) are configured toprocess data that is configured to be rendered in substantial real timeon a display screen. For example, when a user's hand changes positions(e.g., the hand moves, fingers bend, multiple fingers bend, fingerstouch other fingers and/or gestures are made), the changes in positionsare configured to be displayed in substantial real time on a display.

The display may be the display of a head mounted display (HMD), adisplay of a second screen, a display of a portable device, a computerdisplay, a display panel, a display of a remotely connected users (e.g.,whom may be viewing content or sharing in an interactive experience), orthe like. In some embodiments, the captured positions of the user'shand, the pressures sensed, the fingers touched, and/or the hand/fingergestures are used to interact in a video game, in a virtual world scene,a shared virtual space, a video game character, a character that is anextension of the real-world user, or simply provide a way of touching,holding, playing, interfacing or contacting virtual objects shown on adisplay screen or objects associated with documents, text, images, andthe like.

In still other embodiments, virtual gloves may be worn by multiple usersin a multi-user game. In such examples, each user may use one or twogloves. The users may be co-located or interfacing in a shared space orshared game from remote locations using a cloud gaming system, networkeddevice and/or social networked collaboration space. In some embodiments,a glove may be used by one or more remote users to interact in acollaborative way to examine documents, screens, applications, diagrams,business information, or the like. In such an implementation, userscollaborating may use their gloves to touch objects, move objects,interface with surfaces, press on objects, squeeze objects, tossobjects, make gesture actions or motions, or the like.

During collaboration, movements made by one user's hand can appear tothe other user as if a real user hand is moving things, objects, ormaking actions in the collaboration space. Still in a collaborationenvironment, if two remote users are examining documents, users wearinggloves can point at things on a virtual page, point and draw on avirtual whiteboard, lift and move virtual papers, shake hands, moveitems, etc. In some collaborative environments, one or more of the usersmay be wearing an HMD. When the HMD is used in conjunction with theglove or gloves (e.g., worn by one or more users), the users may see avirtual environment in which they can collaborate using their hands,such as moving objects, pages, objects, typing on virtual keyboards,moving virtual pages, tapping on things, pressing on things, etc.

Therefore, it should be understood that the uses of a glove thatincludes one or more sensors, and/or can detect pressure, and/or candetect bending position of fingers, and/or can detect orientation,and/or can detect inertial movement, etc., can provide for a broad scopeof uses. Example uses, without limitation, may include video gaming,entertainment activities, sport related activities, travel and exploringrelated activities, human-to-human contact (e.g., shaking hands of aremote user), business activities, robotic control (e.g. roboticsurgery), etc. In one implementation, this type of interactivityprovided by a glove interface may be extended to additional sensors thatmay be attached or associated with other parts of the human body (e.g.,an arm, a leg, a foot, etc.). In addition to gloves, different types ofclothes are envisioned, e.g., jackets, pants, shoes, hats, etc.

It will be obvious, however, to one skilled in the art, that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known process operations have not beendescribed in detail in order not to unnecessarily obscure the presentinvention.

FIG. 1 illustrates a system for interactive gameplay of a video game, inaccordance with an embodiment of the invention. A user 100 is shownwearing a head-mounted display (HMD) 102. The HMD 102 is worn in amanner similar to glasses, goggles, or a helmet, and is configured todisplay a video game or other content to the user 100. The HMD 102provides a very immersive experience to the user by virtue of itsprovision of display mechanisms in close proximity to the user's eyes.Thus, the HMD 102 can provide display regions to each of the user's eyeswhich occupy large portions or even the entirety of the field of view ofthe user.

In one embodiment, the HMD 102 can be connected to a computer 106. Theconnection to computer 106 can be wired or wireless. The computer 106can be any general or special purpose computer known in the art,including but not limited to, a gaming console, personal computer,laptop, tablet computer, mobile device, cellular phone, tablet, thinclient, set-top box, media streaming device, etc. In one embodiment, thecomputer 106 can be configured to execute a video game, and output thevideo and audio from the video game for rendering by the HMD 102.

The user 100 may operate a glove interface object 104 to provide inputfor the video game. Additionally, a camera 108 can be configured tocapture image of the interactive environment in which the user 100 islocated. These captured images can be analyzed to determine the locationand movements of the user 100, the HMD 102, and the glove interfaceobject 104. In one embodiment, the glove interface object 104 includes alight which can be tracked to determine its location and orientation.Additionally, the HMD 102 may include one or more lights which can betracked to determine the location and orientation of the HMD 102. Thecamera 108 can include one or more microphones to capture sound from theinteractive environment. Sound captured by a microphone array may beprocessed to identify the location of a sound source. Sound from anidentified location can be selectively utilized or processed to theexclusion of other sounds not from the identified location. Furthermore,the camera 108 can be defined to include multiple image capture devices(e.g. stereoscopic pair of cameras), an IR camera, a depth camera, andcombinations thereof.

In another embodiment, the computer 106 functions as a thin client incommunication over a network with a cloud gaming provider 112. The cloudgaming provider 112 maintains and executes the video game being playedby the user 102. The computer 106 transmits inputs from the HMD 102, theglove interface object 104 and the camera 108, to the cloud gamingprovider, which processes the inputs to affect the game state of theexecuting video game. The output from the executing video game, such asvideo data, audio data, and haptic feedback data, is transmitted to thecomputer 106. The computer 106 may further process the data beforetransmission or may directly transmit the data to the relevant devices.For example, video and audio streams are provided to the HMD 102,whereas a vibration feedback command is provided to the glove interfaceobject 104.

In one embodiment, the HMD 102, glove interface object 104, and camera108, may themselves be networked devices that connect to the network 110to communicate with the cloud gaming provider 112. For example, thecomputer 106 may be a local network device, such as a router, that doesnot otherwise perform video game processing, but facilitates passage ofnetwork traffic. The connections to the network by the HMD 102, gloveinterface object 104, and camera 108 may be wired or wireless.

Additionally, though embodiments in the present disclosure may bedescribed with reference to a head-mounted display, it will beappreciated that in other embodiments, non-head mounted displays may besubstituted, including without limitation, a television, projector, LCDdisplay screen, portable device screen (e.g. tablet, smartphone, laptop,etc.) or any other type of display that can be configured to rendervideo and/or provide for display of an interactive scene or virtualenvironment in accordance with the present embodiments.

FIG. 2 illustrates a head-mounted display (HMD), in accordance with anembodiment of the invention. As shown, the HMD 102 includes a pluralityof lights 200A-H. Each of these lights may be configured to havespecific shapes, and can be configured to have the same or differentcolors. The lights 200A, 200B, 200C, and 200D are arranged on the frontsurface of the HMD 102. The lights 200E and 200F are arranged on a sidesurface of the HMD 102. And the lights 200G and 200H are arranged atcorners of the HMD 102, so as to span the front surface and a sidesurface of the HMD 102. It will be appreciated that the lights can beidentified in captured images of an interactive environment in which auser uses the HMD 102. Based on identification and tracking of thelights, the location and orientation of the HMD 102 in the interactiveenvironment can be determined. It will further be appreciated that someof the lights may or may not be visible depending upon the particularorientation of the HMD 102 relative to an image capture device. Also,different portions of lights (e.g. lights 200G and 200H) may be exposedfor image capture depending upon the orientation of the HMD 102 relativeto the image capture device.

In one embodiment, the lights can be configured to indicate a currentstatus of the HMD to others in the vicinity. For example, some or all ofthe lights may be configured to have a certain color arrangement,intensity arrangement, be configured to blink, have a certain on/offconfiguration, or other arrangement indicating a current status of theHMD 102. By way of example, the lights can be configured to displaydifferent configurations during active gameplay of a video game(generally gameplay occurring during an active timeline or within ascene of the game) versus other non-active gameplay aspects of a videogame, such as navigating menu interfaces or configuring game settings(during which the game timeline or scene may be inactive or paused). Thelights might also be configured to indicate relative intensity levels ofgameplay. For example, the intensity of lights, or a rate of blinking,may increase when the intensity of gameplay increases. In this manner, aperson external to the user may view the lights on the HMD 102 andunderstand that the user is actively engaged in intense gameplay, andmay not wish to be disturbed at that moment.

The HMD 102 may additionally include one or more microphones. In theillustrated embodiment, the HMD 102 includes microphones 204A and 204Bdefined on the front surface of the HMD 102, and microphone 204C definedon a side surface of the HMD 102. By utilizing an array of microphones,sound from each of the microphones can be processed to determine thelocation of the sound's source. This information can be utilized invarious ways, including exclusion of unwanted sound sources, associationof a sound source with a visual identification, etc.

The HMD 102 may also include one or more image capture devices. In theillustrated embodiment, the HMD 102 is shown to include image capturedevices 202A and 202B. By utilizing a stereoscopic pair of image capturedevices, three-dimensional (3D) images and video of the environment canbe captured from the perspective of the HMD 102. Such video can bepresented to the user to provide the user with a “video see-through”ability while wearing the HMD 102. That is, though the user cannot seethrough the HMD 102 in a strict sense, the video captured by the imagecapture devices 202A and 202B can nonetheless provide a functionalequivalent of being able to see the environment external to the HMD 102as if looking through the HMD 102. Such video can be augmented withvirtual elements to provide an augmented reality experience, or may becombined or blended with virtual elements in other ways. Though in theillustrated embodiment, two cameras are shown on the front surface ofthe HMD 102, it will be appreciated that there may be any number ofexternally facing cameras installed on the HMD 102, oriented in anydirection. For example, in another embodiment, there may be camerasmounted on the sides of the HMD 102 to provide additional panoramicimage capture of the environment.

FIG. 3 conceptually illustrates the function of the HMD 102 inconjunction with an executing video game, in accordance with anembodiment of the invention. The executing video game is defined by agame engine 320 which receives inputs to update a game state of thevideo game. The game state of the video game can be defined, at least inpart, by values of various parameters of the video game which definevarious aspects of the current gameplay, such as the presence andlocation of objects, the conditions of a virtual environment, thetriggering of events, user profiles, view perspectives, etc.

In the illustrated embodiment, the game engine receives, by way ofexample, controller input 314, audio input 316 and motion input 318. Thecontroller input 314 may be defined from the operation of a gamingcontroller separate from the HMD 102, such as a handheld gamingcontroller (e.g. Sony DUALSHOCK®4 wireless controller, SonyPlaystation®Move motion controller) or glove interface object 104. Byway of example, controller input 314 may include directional inputs,button presses, trigger activation, movements, gestures, or other kindsof inputs processed from the operation of a gaming controller. The audioinput 316 can be processed from a microphone 302 of the HMD 102, or froma microphone included in the image capture device 108 or elsewhere inthe local environment. The motion input 318 can be processed from amotion sensor 300 included in the HMD 102, or from image capture device108 as it captures images of the HMD 102. The game engine 320 receivesinputs which are processed according to the configuration of the gameengine to update the game state of the video game. The game engine 320outputs game state data to various rendering modules which process thegame state data to define content which will be presented to the user.

In the illustrated embodiment, a video rendering module 322 is definedto render a video stream for presentation on the HMD 102. The videostream may be presented by a display/projector mechanism 310, and viewedthrough optics 308 by the eye 306 of the user. An audio rendering module304 is configured to render an audio stream for listening by the user.In one embodiment, the audio stream is output through a speaker 304associated with the HMD 102. It should be appreciated that speaker 304may take the form of an open air speaker, headphones, or any other kindof speaker capable of presenting audio.

In one embodiment, a gaze tracking camera 312 is included in the HMD 102to enable tracking of the gaze of the user. The gaze tracking cameracaptures images of the user's eyes, which are analyzed to determine thegaze direction of the user. In one embodiment, information about thegaze direction of the user can be utilized to affect the videorendering. For example, if a user's eyes are determined to be looking ina specific direction, then the video rendering for that direction can beprioritized or emphasized, such as by providing greater detail or fasterupdates in the region where the user is looking. It should beappreciated that the gaze direction of the user can be defined relativeto the head mounted display, relative to a real environment in which theuser is situated, and/or relative to a virtual environment that is beingrendered on the head mounted display.

Broadly speaking, analysis of images captured by the gaze trackingcamera 312, when considered alone, provides for a gaze direction of theuser relative to the HMD 102. However, when considered in combinationwith the tracked location and orientation of the HMD 102, a real-worldgaze direction of the user can be determined, as the location andorientation of the HMD 102 is synonymous with the location andorientation of the user's head. That is, the real-world gaze directionof the user can be determined from tracking the positional movements ofthe user's eyes and tracking the location and orientation of the HMD102. When a view of a virtual environment is rendered on the HMD 102,the real-world gaze direction of the user can be applied to determine avirtual world gaze direction of the user in the virtual environment.

Additionally, a tactile feedback module 326 is configured to providesignals to tactile feedback hardware included in either the HMD 102 oranother device operated by the user, such as a controller 104. Thetactile feedback may take the form of various kinds of tactilesensations, such as vibration feedback, temperature feedback, pressurefeedback, etc.

As has been noted, the HMD device described herein is capable ofproviding a user with a highly immersive experience, enveloping a largeproportion or even an entirety of a user's field of vision. In light ofthis immersive aspect of the HMD experience, it is desirable to provideintuitive control mechanisms to the user, especially as the user may notbe able to see their own hands or objects (e.g. controller) they areholding. Thus, in accordance with embodiments of the invention describedherein, methods, apparatus, and systems are provided for a gloveinterface object.

Throughout the present disclosure, reference is made to the gloveinterface object and the user's hand, including the fingers, palm, andother portions thereof. For purposes of ease of description andreadability of the present disclosure, it will be understood by thoseskilled in the art that the glove interface object and the user's hand(and/or portion thereof) may in many instances be referencedinterchangeably and/or in the alternative. That is, an activity (e.g.pose, position, movement, orientation, location, action, etc.) definedby a user's hand, also pertains to the glove interface object that isbeing worn on the user's hand, as the glove interface object isconfigured to detect or facilitate detection of the activity of theuser's hand. Therefore, it may be convenient for descriptive purposes todiscuss certain aspects in the present disclosure utilizing languagepertaining to the user's hand. However, it will be readily appreciatedthat the glove interface object is worn on the user's hand and that suchmay apply or in fact be defined by the glove interface object, thisbeing apparent to those skilled in the art from the context of thedescription.

FIG. 4A illustrates a glove interface object having a plurality ofemitters and proximity sensors defined thereon, in accordance with anembodiment of the invention. As shown, the glove interface object 400includes various finger portions, including a thumb portion 402 a, andindex finger portion 402 b, a middle finger portion 402 c, a ring fingerportion 402 d, and a pinky/little finger portion 402 e. A plurality ofproximity sensors are defined substantially at the fingertipportions/areas of the glove interface object 400. That is, the proximitysensors are defined on/in or otherwise at or near the end portions ofthe finger portions of the glove interface object 400 that correspond toor receive the distal phalanges portions of the user's fingers. In someimplementations, the fingertip portions/areas correspond to some distalportion (e.g. distal half, distal third, distal quarter, etc.) of thedistal phalanges, whereas in other implementations, the fingertipportions/areas correspond to the entire distal phalanges. The fingertipportion/area of a finger may include the top, bottom, sides, and/or endof the finger, and/or any sub-combination thereof. In the illustratedembodiment, proximity sensors 404 a, 404 b, 404 c, 404 d, and 404 e arerespectively defined at the fingertips/ends of the thumb portion 402 a,index finger portion 402 b, middle finger portion 402 c, ring fingerportion 402 d, and little finger portion 402 e. Additionally, aproximity sensor 406 is defined on a back portion or a palm portion ofthe glove interface object 400. The proximity sensors are configured togenerate data indicating distance/proximity to each of a plurality ofemitters 422 a, 422 b, and 422 c.

In the illustrated implementation, the emitters are defined at a wristportion 424 of the glove interface object 400. In some implementations,the wrist portion 424 is defined as a bracelet that surrounds the user'swrist when the glove interface object 400 is worn. The wrist portion 424is configured to remain substantially stationary (e.g. have a stableorientation) with respect to the user's wrist/forearm, even as theremainder of the glove interface object 400, such as the finger portionsand the palm portion, are moved in accordance with movements of theuser's hand wearing the glove interface object 400. By maintaining thewrist portion 424 in a substantially stable orientation with respect tothe user's wrist/forearm, then the emitters will also be maintained in asubstantially stable orientation relative to the wrist/forearm, so thatthe proximity of the fingertip portions relative to the emitters can beconsistently tracked and reliably indicate the proximity of the user'sfingertips to his/her wrist/forearm.

In some implementations, the wrist portion 424 is configured to bepositioned on the distal portion of the user's forearm. As used herein,the term “wrist” can include the distal portion of the forearm, inaddition to the wrist joints which comprise the central part of thewrist. While the distal portion of the forearm is subject to pronationand supination movements, it is not subject to the other movements ofthe hand facilitated by the wrist joints, such as marginal movementsincluding radial deviation and ulnar deviation, or flexion movementssuch as palmar flexion and dorsiflexion/extension. As such, by securingthe wrist portion 424 to the distal portion of the forearm, it ispossible for the emitters which are defined at the wrist portion 424 toserve as reference locations facilitating detection of changes resultingfrom the aforementioned movements of the hand which are facilitated bythe wrist joints.

In various implementations, the wrist portion 424 can be secured to thewrist/forearm of the user (e.g. the distal end/portion of the forearm)by any known device, construction, or method which provides for secureplacement of the wrist portion 424 so that the location/orientation ofthe wrist portion 424 is substantially unaffected by movements of theuser's hand/fingers relative to the distal end/portion of the user'sforearm. In some embodiments, the wrist portion 424 includes an elasticband for securing the wrist portion to the user's wrist/forearm. In someembodiments, the wrist portion 424 includes a clasp, buckle, tie, strap,or other mechanism for securing the wrist portion 424 to the user'swrist/forearm. In some embodiments, the mechanism can be adjustable toaccommodate different sized wrists/forearms. In some embodiments, theinterior surface of the wrist portion 424 is defined from a materialproviding for friction against the skin of the user's wrist/forearm toprevent the wrist portion 424 from slipping. In some embodiments, theinterior surface may include a three-dimensional surface structure orpatterning that is configured to prevent slippage, such as ribbing,dimples, etc.

In some implementations, the emitters are defined by electromagnets, andthe proximity sensors are defined by magnetic sensors such as Halleffect sensors. Broadly speaking, a Hall effect sensor is a transducerthat varies its output voltage in response to a magnetic field. A Halleffect sensor may consist of a rectangular p-type semiconductor material(e.g. gallium arsenide, indium antimonide, indium arsenide) throughwhich a current is passed. When the sensor is placed within a magneticfield, the magnetic flux lines exert a force which deflects the chargecarriers (electrons and holes) to either side of the semiconductor slab.The movement of charge carriers, resulting from the magnetic force theyexperience passing through the semiconductor material, causes a build-upof charge carriers producing a potential difference between the twosides of the semiconductor material. The output voltage (Hall voltage)is proportional to the strength of the magnetic field passing throughthe semiconductor material. A Hall effect sensor is one example of amagnetic sensor that may be utilized to detect a magnetic field. Inother embodiments, other types of magnetic sensors which are capable ofdetecting and measuring a magnetic field may be utilized.

In some implementations, the emitters are defined by ultrasonic emittersand the proximity sensors are defined by microphones capable ofdetecting ultrasonic frequencies. In some implementations, the emittersare defined by RF emitters and the proximity sensors are defined by RFdetectors. It will be appreciated that in various embodiments, theemitters and proximity sensors can be defined by any combination ofemitters that emit signals and proximity sensors capable of detectingsaid signals, wherein the proximity sensors generate data based on thedetected signals that indicates the proximity/distance of the proximitysensors to the emitters. Broadly speaking, embodiments discussed hereinare described in terms of emitters that are defined by electromagnetsand proximity sensors that are defined by magnetic sensors such as Halleffect sensors. However, it will be appreciated that in otherembodiments, other types of emitters and proximity sensors may besubstituted to achieve the same or similar functionality withoutdeparting from the scope of the present disclosure.

The glove interface object 400 includes a controller 408 that isconfigured to control the operation of the proximity sensors and theemitters. In the illustrated embodiment, the controller 408 is definedas part of the wrist portion 424, though in other embodiments, thecontroller 408 (or any of its specific subcomponents) can be located atdifferent locations on or in the glove interface object 400. In oneembodiment, the controller 408 includes various components, such as apower source 410 for providing power to operate the controller as wellas the proximity sensors and emitters, a wired/wireless transceiver 412for transmitting and receiving data with an external computing devicesuch as a gaming console, a processor 414 for executing programinstructions, a memory 416 for storing data and program instructions, anemitter controller 418 for controlling the operation of the emitters,and a sensor controller 420 for controlling the operation of theproximity sensors.

In some implementations, the controller 408 is configured to control theactivation of the emitters and the reading of the proximity sensors in atime division multiplexed arrangement. That is, the emitters areserially activated and deactivated to define separate periods ofactivation for each emitter. During each period of activation for agiven emitter, each of the proximity sensors can be read to obtainsensor data indicating the proximity/distance of each proximity sensorto the given emitter that is currently activated. The time divisionmultiplexed arrangement will be more clearly understood throughconsideration of a specific example below.

With continued reference to FIG. 4A, in some implementations, thefollowing time division multiplexed arrangement for control of theemitters and proximity sensors is utilized. During a first time period,the first emitter 422 a is activated, and each of the proximity sensorsis read; during a second time period that is successive to the firsttime period, the second emitter 422 b is activated, and each of theproximity sensors is read a second time; and during a third time periodthat is successive to the second time period, the third emitter 422 c isactivated, and each of the proximity sensors is read a third time. Thissequence of activation of the emitters can be repeated in a cyclicalfashion to define a cycle/sequence of activation periods of theemitters. It will be understood that in some implementations, eachactivation time period for a given emitter is substantially exclusive ofother activation time periods for the other emitters. In other words,the activation time periods for the emitters do not substantiallyoverlap with one another. However, it will be appreciated that in someembodiments, there may be an overlap between the activations of theemitters. For example, as one emitter is being deactivated, the nextemitter may be simultaneously activated, and thus an overlap between theactivation time periods of both emitters may exist (e.g. possibly due toinductance in coils of the electromagnets). In some implementations,such an arrangement can be configured to be substantiallynonoverlapping, wherein the time when a given emitter is exclusivelyactivated is substantially greater than the time that it overlaps withanother emitter, and the reading of the proximity sensors occurs duringthis exclusive activation time. In other implementations, thearrangement may be configured to be substantially overlapping. It willbe appreciated that in various embodiments, the amount of overlap mayvary, provided that the sequence of activation of the emitters provideseach of the emitters to be exclusively activated at different times, asthe proximity sensors are read during the exclusive activation times ofeach emitter.

For example, in implementations wherein the emitters compriseelectromagnets and the proximity sensors comprise magnetic sensors suchas Hall effect sensors, then the deactivation of one electromagnet maycoincide with or overlap with the simultaneous activation of anotherelectromagnet. In this manner, time is saved during theactivation/deactivation times of the electromagnets, so as to minimizethe amount of time spent transitioning from an activated state of oneelectromagnet to an activated state of a next electromagnet. It will beappreciated that in such implementations, it is important to provide forperiods of activation of each electromagnet that are exclusive of eachother so that when the Hall effect sensors are read during such periodsof activation, the magnetic fields of each electromagnet do notinterfere substantially with one another.

In some implementations, the emitters have a non-collinear arrangement.That is, not all of the emitters are defined substantially along asingle line. By way of example, with continued reference to FIG. 4A, inone embodiment, the emitters are arranged on the wrist portion 424 ofthe glove interface object 400 so as to be positioned in a configurationsurrounding the wrist of the user. Such an arrangement is shown withreference to the cross-sectional view A, showing the emitters 422 a, 422b, and 422 c disposed in the wrist portion 424 in a non-collineararrangement surrounding the wrist of the user when the glove interfaceobject 400 is worn on the user's hand.

In some implementations, there are at least three emitters that arearranged in a non-collinear arrangement. By having at least threeemitters in a non-collinear arrangement, then by determining distancesfrom a given proximity sensor to each of the emitters, it is thenpossible to determine the specific location of the given proximitysensor relative to the emitters. The specific calculations which arerequired to determine the location of a given proximity sensor relativeto the emitters will be apparent to those skilled in the art, and aretherefore not discussed here in detail. However, broadly speaking, sucha determination entails determining an intersection of line segmentsdefined from each of the emitters, wherein each line segment has oneendpoint defined at one of the emitters and another endpoint defined atthe intersection. The line segments have lengths that are defined by theproximity/distance of the given proximity sensor from each of theemitters, as determined from data generated by the given proximitysensor. Described another way, the location of the given proximitysensor relative to the emitters is defined by the intersection of radiihaving origins defined by the locations of the emitters and magnitudesdefined by the distances of each of the emitters to the given proximitysensor.

By way of example, with reference to FIG. 4A, the location of the indexfinger proximity sensor 404 b that is located at the end of the indexfinger portion 402 b of the glove interface object 400 can be defined inthe following manner. During the periods of activation of each of theemitters, the index finger proximity sensor 404 b is read and thedistance from the proximity sensor 404B to each of the emitters can bedetermined. A first radius can be defined having an origin located atthe first emitter 422 a and a magnitude defined by the distance from theindex finger proximity sensor 404 b to the first emitter 422 a. A secondradius can be defined having an origin located at the second emitter 422b and a magnitude defined by the distance from the index fingerproximity sensor 404 b to the second emitter 422 b. A third radius canbe defined having an origin located at the third emitter 422 c and amagnitude defined by the distance from the index finger proximity sensor404 b to the third emitter 422 c. Based on this information, thelocation of the index finger proximity sensor relative to the emitterscan be defined as the intersection of the first, second, and thirdradii. It will be apparent to those skilled in the art that a similarprocedure can be applied to determine the location of the otherproximity sensors relative to the emitters.

As is shown in the illustrated embodiment, proximity sensors areprovided at the fingertips of the glove interface object. In accordancewith the principles discussed, each of these proximity sensors can beutilized to determine the location of the fingertips of the gloveinterface object relative to the emitters. It will be appreciated thatin other implementations, additional proximity sensors may be located onor within the glove interface object, and that such may be utilized todetermine the specific locations of particular parts of the gloveinterface object relative to the emitters. By way of example, aproximity sensor 406 is shown positioned on either a backhand portion ora palm portion of the glove interface object. The proximity sensor 406can be utilized to determine the location of the backhand portion or thepalm portion of the glove interface object, and thereby also indicateflexion and deviation of the user's hand as facilitated by bendingmovements of the user's wrist.

FIG. 4B illustrates a glove interface object having multiple proximitysensors defined thereon, in accordance with an embodiment of theinvention. In the illustrated embodiment, in addition to the proximitysensors 404 a-e which are defined at fingertip portions of the gloveinterface object 400, there are proximity sensors positioned along thefinger portions of the glove interface object to coincide with thejoints of the fingers of the user's hand when wearing the gloveinterface object. These include proximity sensors 430 a-d which arepositioned to coincide with the distal interphalangeal joints, proximitysensors 432 a-e which are positioned to coincide with the proximalinterphalangeal joints, and proximity sensors 434 a-e which arepositioned to coincide with the metacarpophalangeal joints. By providingproximity sensors that coincide with the joints of the user's hand whenwearing the glove interface object 400, then the pose of the user's handcan be determined with a high degree of precision.

FIG. 5 is a diagram conceptually illustrating the operation of severalemitters and sensors of a glove interface object in a time divisionmultiplexed arrangement, in accordance with an embodiment of theinvention. For purposes of discussing the instant embodiment, theemitters consist of a plurality of electromagnets M1, M2, and M3.However, in other embodiments, other types of emitters may be utilizedto achieve similar effects. Similarly, in the instant embodiment, aplurality of magnetic sensors S1, S2, S3, S4, and S5 are defined onfinger portions (e.g. at the fingertips) of the glove interface object.In other embodiments, there may be any type of sensor capable ofdetecting emissions from the emitter.

During a first time period T1, electromagnet M1 is exclusivelyactivated. That is, during the first time period T1, electromagnet M1 isin an activated state while electromagnets M2 and M3 are in adeactivated state. During this time period of activation T1, each ofsensors S1, S2, S3, S4, and S5 is read. During a second time period T2,which succeeds the first time period T1, electromagnet M2 is exclusivelyactivated, such that electromagnet M2 is in an activated state whileelectromagnets M1 and M3 are in a deactivated state. During the secondtime period T2, each of sensors S1, S2, S3, S4, and S5 are read. Duringa third time period T3, which succeeds the second time period T2,electromagnet M3 is exclusively activated, such that electromagnet M3 isin an activated state while electromagnets M1 and M2 are in adeactivated state. During the third time period T3, each of the sensorsS1, S2, S3, S4, and S5 are read.

At the conclusion of one cycle of time periods T1, T2, and T3, each ofthe sensors will have been read once during the activation of eachelectromagnet (i.e. in the present example, each sensor is read threetimes, in respective correspondence to the three electromagnets). Insome implementations, each sensor is configured to detect the strengthof a magnetic field generated by the electromagnets. As the strength ofa magnetic field varies with distance, then the detected strength of themagnetic field can indicate the distance of a given sensor to theelectromagnet. Thus, for each sensor, the three readings taken per cycleindicate the distances of the sensor to each of the electromagnets. Andas has been discussed, based on these distances, a relative location ofthe given sensor to the electromagnets can be determined. It will beappreciated that in the illustrated embodiment, three electromagnets andfive sensors are shown; however, in other embodiments, there may be morethan three electromagnets and more than five sensors.

Thus in accordance with the foregoing, the relative locations of each ofthe sensors to the electromagnets can be determined. In implementationswherein the sensors are positioned at the fingertips of the gloveinterface object, then the locations of the fingertips of the gloveinterface object can thus be determined. The successive activation ofeach of the electromagnets defines a cycle of activation of theelectromagnets which can be repeated. It will be appreciated that duringeach cycle of activation of the electromagnets, the locations of thesensors (and by extension, the locations of the user's fingertips) areupdated. In this manner, by generating a repeated cycle of activation ofthe electromagnets in combination with reading of the sensors during theperiods of activation of the electromagnets, the locations of thesensors can be continually tracked.

It will be appreciated that as the locations of the sensors arecontinually tracked, then the real-time relative location of the user'sfingers can be provided. In some implementations, a virtual hand in avirtual environment is controlled so as to have a pose that tracks thatof the user's hand as determined from the tracking of the sensors on theglove interface object. This provides for a user to have real-timecontrol of the pose of the virtual hand, as the movements of the user'shand/fingers will be substantially replicated by the virtual hand inreal-time.

In some implementations, it can be desirable to determine and accountfor the presence of ambient magnetic fields that may be produced byother sources of magnetic fields in the local environment. Therefore,with continued reference to FIG. 5, in one embodiment during a timeperiod T4, each of the sensors is read while each of the electromagnetsis deactivated. The readings of the sensors taken while theelectromagnets are deactivated indicate the strengths of magnetic fieldswhich may be present in the local environment. Thus, these readings canbe utilized to account for such ambient magnetic fields when determiningthe locations of the sensors relative to the electromagnets of the gloveinterface object, which is based on reading the sensors during theactivation periods of the electromagnets. For example, for a givensensor, the ambient reading could be subtracted from a reading takenduring activation of one of the electromagnets.

In some embodiments, the ambient sensor readings (i.e. readings of oneor more of the sensors when the electromagnets are deactivated) aretaken once for each cycle of activation of the electromagnets. Howeverin other embodiments, the ambient sensor readings may be taken onceevery N number of cycles of activation of the electromagnets, wherein Nis any natural number. By taking ambient sensor readings at a rate thatis less than the rate of activation of each of the electromagnets, thenless processing resources are consumed for the purpose of ambientreadings than if ambient sensor readings are taken at the same rate;however, the fidelity of tracking the ambient readings is reduced.

In some implementations, the frequency of ambient readings can bepositively correlated to an amount of movement of the glove interfaceobject as a whole, such that ambient readings are taken with higherfrequency when the glove interface object is determined to be moving ata higher rate, and ambient readings are taken with lower frequency whenthe glove interface object is determined to be moving at a lower rate.For example, in some implementations, ambient readings will cease to betaken when the glove interface object is determined to be in astationary location for a predetermined amount of time. It will beappreciated that the location of the glove interface object can bedetermined based on data from motion sensors included in the gloveinterface object, from visual tracking of the glove interface object ora specific part thereon, or from any other method of tracking thelocation of the glove interface object.

In the foregoing implementations, it has generally been described thatduring each period of activation for each of the electromagnets, each ofthe sensors is read once. However, in other implementations, not everysensor is necessarily read during the period of activation of each ofthe electromagnets. In other words, though in some embodiments a givensensor is read during each cycle of activation of the electromagnets, inother embodiments, the given sensor may not be read during each cycle ofactivation of the electromagnets. That is, the given sensor may be readduring every Nth cycle of activation, wherein N is any natural number.By selectively reading the sensors during particular cycles ofactivation of the electromagnets, particular sensors (and by extension,certain fingertips of the user) can be prioritized for tracking overothers. For example, it may be the case that for a particular video gameactivity, movements of an index finger are more important than movementsof the thumb. It may therefore be desirable to take readings of theindex finger sensor with greater frequency (i.e. during more cycles ofactivation for a given unit of time) than the thumb sensor. In thismanner, system resources can be prioritized to provide for greaterfidelity tracking of the index finger over that of the thumb.

Extending the concept further, the specific rate of reading any of thesensors can be dynamically adjusted by the system in accordance with thecontext of the interactive application (e.g. video game) with which theglove interface object is being used. For example, an executing videogame may set and dynamically adjust a parameter that defines the rate(per cycles of activation) at which a given sensor is read.

Depending upon the processing capabilities of the hardware of the system(e.g. the glove interface object and a computing device to which theglove interface object sends data), the readings of the sensors can beprocessed in sequential order (e.g. some or all of the sensors are readin predefined order during a given period of activation of anelectromagnet) or in parallel (e.g. some or all of the sensors are readsimultaneously during a given period of activation of an electromagnet).For example, in some implementations, the glove interface object mayinclude a multi-channel sampling capability, so that some or all of itssensors can be read simultaneously. Subsequently, the processing of thereadings may be performed in parallel and/or sequential manner todetermine and track the locations of the sensors relative to theelectromagnets.

It will be appreciated that the time-division multiplexing arrangementfor activation of electromagnets and readings of the sensors (andassociated processing to determine and track locations of parts of theglove interface object such as the fingertips) can be extended to thescenario where multiple glove interface objects are being utilized (e.g.left and right gloves for each hand of a single user, left and/or rightgloves for each of multiple users). For example, the cycles ofactivation of the electromagnets for a left hand glove interface object(and its accompanying sensor readings) may alternate with the cycles ofactivation of the electromagnets for a right hand glove interface object(and its accompanying sensor readings).

In some embodiments, the cycles of activation of the electromagnets foreach of multiple gloves (and the accompanying sensor readings) arethemselves performed in a cyclical manner, so that a cycle of activationfor a first glove interface object is followed by a cycle of activationfor a second glove interface object, which is followed by a cycle ofactivation for a third glove interface object, etc., returning to thefirst glove interface object to repeat the overall sequence, and therebyfacilitating tracking of the fingertips or other parts of the gloveinterface object.

Though in the present disclosure, embodiments are described withreference to electromagnets situated at or near a user's wrist, itshould be appreciated that in other embodiments, such electromagnets maybe located at other locations, such as on or near the user's palm, backof the palm, forearm, torso, or other body part, on the HMD, etc. Theelectromagnets may be arranged at any location that provides for thefunctionality described in accordance with the embodiments discussedherein.

FIG. 6 illustrates several graphs showing the application of power tovarious electromagnets of a glove interface object, in accordance withan embodiment of the invention. Graph 600 illustrates the magneticfields produced by each of several electromagnets M1, M2, and M3. Asshown the magnetic fields for each of the electromagnets define periodsduring which each of the electromagnets is exclusively activated.However, an overlap exists during the deactivation of one electromagnetand the activation of a next electromagnet (e.g. when M1 is beingdeactivated and M2 is being activated). Also noted on the graph 600 aresampling times ST1, ST2, ST3, ST4, ST5, and ST6, which are the timepoints at which the sensors of the glove interface object are read. Inthe illustrated embodiment, the sensors are read at time points thatcorrespond to the magnetic fields reaching substantially peak strengths.However, in other embodiments, the sensors may be read at time pointscorresponding to the magnetic fields reaching other relative strengths,provided the strengths of the magnetic fields when the sensors are readare consistent. That is, each time the sensors are read during theactivation of the electromagnet M1, the strength of the electromagnet M1is the same.

The graphs 602, 604, and 606 illustrate the voltage applied for each ofthe electromagnets M1, M2, and M3, respectively. As shown, in someimplementations, a specific voltage (e.g. positive voltage) is appliedto a given electromagnet (e.g. M1) that causes the magnetic field toincrease. After a specified amount of time, the specific voltage may bereversed (e.g. negative voltage) so as to accelerate the reduction ofthe magnetic field and/or counteract any induced/temporary magnetizationeffects (or latent magnetic fields). Latent magnetic fields may resultfrom inductance of the electromagnet coils, and as such, diodes may beused to more quickly effect changes in the magnetic field.

FIG. 7 illustrates a glove interface object configured for use with aperipheral device, in accordance with an embodiment of the invention. Inthe illustrated embodiment, the glove interface object includes emitters702 a, 702 b, and 702 c, which are defined on a wrist portion of theglove interface object. A plurality of sensors 704 a, 704 b, 704 c, 704d, and 704 e are defined at fingertip portions of the glove interfaceobject 700. Also shown is a peripheral device 710 that may be held orcontacted by the glove interface object. In the illustrated embodiment,the peripheral device 710 is a motion controller including anilluminated object 712 that can be illuminated to facilitate trackingbased on analysis of captured image data. Further, the peripheral device710 may include various kinds of input devices such as buttons 714 forproviding input, as well as various types of motion sensors, such asaccelerometers, magnetometers, gyroscopes, etc. It will be appreciatedthat though a specific motion controller is shown, in otherimplementations, the peripheral device 710 may be any other type ofinteractive equipment utilized to provide input for an interactiveapplication in accordance with the principles discussed herein. Theperipheral device may communicate wirelessly with a computing devicesuch as a gaming console.

The peripheral device may also include emitters 716 a, 716 b, and 716 c,which can be sensed by the sensors on the glove interface object 700.The peripheral emitters 716 a, 716 b, and 716 c can be operated in asimilar or the same manner as that described with respect to emitters onthe glove interface object. More specifically, the sensors can detectthe strengths of signals emitted by the peripheral emitters, andgenerate data indicating proximity/distance of the sensors to theperipheral emitters. This information can be processed to identify theorientation/configuration of the user's hand relative to the peripheraldevice.

Furthermore, in various implementations, the operation of the gloveemitters (702 a, 702 b, and 702 c) and peripheral emitters (716 a, 716b, and 716 c), in combination with operation of the sensors (704 a, 704b, 704 c, 704 d, and 704 e) can have various configurations. Generallyspeaking, some or all of the sensors are read during the activation of aspecific emitter on either the glove interface object or the peripheraldevice. Some possible configurations regarding the activation of theemitters are discussed below.

In some implementations, the glove emitters and the peripheral emittersare utilized in combination to allow tracking of the sensor locationsrelative to both of the glove emitters and the peripheral emitters. Insome implementations, this may entail time-division multiplexing theactivation sequences of the glove and peripheral emitters with eachother. For example, activation of a glove emitter may alternate withactivation of a peripheral emitter so that the activation sequences ofthe glove and peripheral emitters are interwoven with each other. Inanother embodiment, each of the glove emitters and peripheral emittersmay alternate activation sequences.

In some implementations, the locations of the glove interface object andthe peripheral object in space are tracked, and the activation of theemitters on both devices is determined and controlled based on theirlocations in space. For example, in one embodiment, when the gloveinterface object and the peripheral device are separated by greater thana predefined distance, then the glove emitters are utilized, while theperipheral emitters are deactivated and not utilized. However, when theseparation between the glove interface object and the peripheral devicereaches or becomes less than the predefined distance, then the gloveemitters are deactivated and no longer utilized, while the peripheralemitters are activated and utilized. Thus, the sensors transition fromsensing proximity to the glove emitters to sensing proximity to theperipheral emitters. It will be appreciated that transitioning from oneset of emitters to the other may provide further advantages by reducingthe number of emitters that are multiplexed (as compared tosimultaneously using both sets of emitters) and reducing powerconsumption, e.g. of the glove interface object when the glove emittersare deactivated.

In another implementation, the utilization of the glove emitters versusthe peripheral emitters may be controlled by an interactive applicationor video game for which the devices are providing input. For example,during one portion of a video game, the peripheral device is notutilized for input, and hence the peripheral emitters are deactivatedwhile the glove emitters are activated and utilized. Whereas duringanother portion of the video game, the peripheral device is utilized forinput, and hence the glove emitters will be deactivated while theperipheral emitters are activated and utilized.

When a user is using a head-mounted display (HMD), the user may not havethe ability to see the local environment external to the HMD. However,by utilizing sensors on the glove interface object to detect emitters onthe peripheral device, the location of the peripheral device relative tothe glove interface object can be determined. Furthermore, theconfiguration of the user's hand and the peripheral device can bedefined with precision to, for example, define the configuration of acorresponding virtual hand and virtual object that exist in a virtualspace that is rendered on the HMD. For example, the peripheral devicecould define the positioning of the virtual object in the virtual space(e.g. a weapon such as a sword or gun), and the virtual hand could beshown holding the virtual object in a manner that is similar to theuser's hold on the peripheral device. The movements of the user'sfingers in relation to the peripheral device could be tracked andrendered as corresponding movements of the fingers of the virtual handin relation to the virtual object.

FIG. 8 illustrates a pair of glove interface objects configured tointeract with a keyboard peripheral device, in accordance with anembodiment of the invention. A right hand glove interface object 800 isshown, including proximity sensors 802 a-e; a left hand glove interfaceobject 804 includes proximity sensors 806 a-e. The proximity sensors 802a-e and 806 a-e can be configured to detect emitters defined on theirrespective glove interface objects (e.g. defined on wrist portions ofthe glove interface objects).

A keyboard peripheral device 810 may include various keys 812 forproviding input to an interactive application. The keyboard 810 maycommunicate wirelessly with a computing device that executes theinteractive application. The keyboard 810 further includes emitters 814a-c, which can be detected by the proximity sensors 802 a-e and 806 a-e.In some implementations, the emitters 814 a-c are electromagnets thatproduce magnetic fields, and the proximity sensors are magnetic sensors(e.g. Hall effect sensors) that detect the magnetic fields. Theactivation of the emitters and the reading of the proximity sensors canbe configured in a time division multiplexed arrangement, as discussedpreviously. With such an arrangement, the locations of the proximitysensors relative to the emitters can be determined and tracked. And byextension, in implementations wherein the proximity sensors arepositioned at the fingertip portions of the glove interface object, thenthe locations of the fingertips relative to the keyboard peripheraldevice can be determined and tracked.

The above-described configuration wherein proximity sensors on gloveinterface objects are utilized to detect emitters positioned on aperipheral device allows for very precise tracking of the user's handsin relation to a peripheral device. Thus, by way of example, virtualhands corresponding to the user's hands can be shown in a virtual spaceinteracting with a virtual keyboard that corresponds to the keyboardperipheral device 810 with a high degree of fidelity. Though in theillustrated embodiment, a keyboard is specifically shown, it should beappreciated that in various other embodiments any kind of peripheraldevice can be configured to provide similar functionality as thatdescribed herein.

In another embodiment, the peripheral device 810 may not specificallyinclude keys of a keyboard, but may define an interactive surface withwhich the user's hands may interact. The relative locations of theuser's hands (including the fingertips) to the interactive surface ofthe peripheral device can be determined and tracked in accordance withthe above-described methods. Furthermore, the peripheral device candefine a virtual object in a virtual space, and virtual handscorresponding to the user's hands can be shown interacting with thevirtual object in a manner that is defined by the interaction of theglove interface objects with the peripheral device. It will beappreciated that the virtual object can be defined to have anyconfiguration and may include any sub objects without limitation. Forexample, the virtual object may impact define a virtual keyboard, andthe virtual hands can be shown interacting with the virtual keyboard, asdefined by the movements of the fingertips of the glove interfaceobjects in relation to the peripheral device 810.

The virtual object can have a virtual surface corresponding to theinteractive surface of the peripheral device, and interactions of thevirtual hands with the virtual surface can be defined by theinteractions of the glove interface objects with the interactive surfaceof the peripheral device. This type of arrangement can facilitateinteraction with virtual objects by a user in an intuitive manner. Forexample, the virtual object may be a device having a touchscreeninterface, and the user may thus interact with the virtual touchscreenby interacting with the interactive surface of the peripheral device. Itshould be appreciated that various kinds of interactions with theinteractive surface of the peripheral device can be detected andrecognized, including touches, taps, swipes, gestures, multi-fingertouches/gestures, etc. As another example, the virtual object mayinclude input devices, and the user may interact with the virtual inputdevices by interacting with the peripheral device.

In some implementations, the peripheral device 810 may additionallyinclude long-range emitters 816 a-c. The long-range emitters areconfigured to provide stronger signals than the emitters 814 a-c, andare therefore detectable by the proximity sensors of the glove interfaceobjects at a greater distance. In some implementations, the long-rangeemitters are utilized when the proximity sensors of the glove interfaceobjects are located at a distance greater than a predefined threshold,whereas the emitters 814 a-c are utilized when the proximity sensors ofthe glove interface objects are located at a distance at or less thanthe predefined threshold. When the glove interface objects aredetermined to have moved from being located beyond the threshold towithin the threshold, then the system may transition from using thelong-range emitters 816 a-c to using the regular emitters 814 a-c.

In some implementations, the peripheral device 810 may include anilluminated object 818, which can be illuminated for purposes of visualtracking. That is, the illuminated object can be recognized based onanalysis of captured images of the interactive environment in which theperipheral device is located, and the location and/or orientation of theperipheral device can therefore be determined and tracked.

FIG. 9 illustrates a peripheral device having long-range and close rangeemitters configured to provide for course tracking and fine tracking ofone or more glove interface objects having proximity sensors, inaccordance with an embodiment of the invention. In implementationsdiscussed below, the emitters are defined as electromagnets and theproximity sensors are magnetic sensors such as Hall effect sensors;however, in other implementations, other types of emitters andcorresponding proximity sensors can be utilized. As shown, the device900 includes long-range electromagnets (emitters) 902 a-d andclose-range electromagnets (emitters) 904 a-c. The long-range andclose-range electromagnets can be activated to define magnetic fieldsthat are sensed by magnetic sensors of a glove interface object, toprovide for determination and tracking of the location and/ororientation of the glove interface object (e.g. the fingertips) relativeto the device 900. The use of the electromagnets and magnetic sensors todetermine the relative location of the glove interface object to thedevice 900 can be performed in accordance with the principles ofoperation previously described, including time-division multiplexedarrangements for controlling the electromagnets and reading the sensors.

In some implementations, the activation of the long-range andclose-range electromagnets is dependent upon distance. For example, inone embodiment, when the location of the glove interface object (e.g. asdefined by the location of magnetic sensors defined thereon) exceeds apredefined distance threshold 910 from the device 900, then theclose-range electromagnets 904 a-c are deactivated while the long-rangeelectromagnets 902 a-d are activated. Whereas when the location of theglove interface object is within the predefined distance threshold 910,then the close-range electromagnets are activated while the long-rangeelectromagnets are deactivated. In some implementations, when the gloveinterface object reaches the threshold 910, then the system transitionsfrom using either of the close/long range electromagnets to using theother set of electromagnets, the sensor readings/data transition fromindicating proximity/distance to one set of electromagnets to indicatingproximity/distance to the other set of electromagnets. In someimplementations, the transition occurs after the glove interface object(and the proximity sensors) crosses over the threshold 910 for apredefined amount of time or predefined number of cycles of sensorsampling or electromagnet activation.

In some implementations, the tracking precision is adjusted dependingupon distance of the proximity sensors (or the glove interface object)from the electromagnets. For example, in some implementations, thesample rate of the proximity sensors and the corresponding frequency ofactivation sequences of the electromagnets (e.g. electromagnets 904 a-cand/or 902 a-d) decreases as the distance of the proximity sensors fromthe electromagnets increases. In this manner, as the glove interfaceobject approaches the device 900, then the tracking becomes more preciseand fine-grained; and as the glove interface object moves away from thedevice 900, then the tracking becomes more course, and utilizes lessbandwidth and processing resources.

It will be appreciated that the above-described variation in trackingprecision according to distance from the electromagnets (or the device900) can be applied in combination with the long-range and close-rangeelectromagnets. For example, in one embodiment, when the glove interfaceobject is within the threshold distance 910 from the device 900, and theclose-range electromagnets are actively utilized, then the sampling rateof the proximity sensors and the corresponding frequency of theactivation sequences of the close-range electromagnets are set at afirst level. Whereas when the glove interface object exceeds thethreshold distance 910 from the device 900, and the long-rangeelectromagnets are actively utilized, then the sampling rate of theproximity sensors and the corresponding frequency of the activationsequences of the close-range electromagnets are set at a second levelthat is less than the first level. In this manner, then the region thatis at or within the threshold distance 910 defines a fine trackingregion, and the region that is beyond the threshold distance 910 definesa course tracking region.

In some implementations, another distance threshold 912 is definedbeyond which no electromagnets of the device 900 are activated.Therefore, in such implementations, the course tracking region isbounded by the distance threshold 910 and the distance threshold 912.When the glove interface object exceeds the distance threshold 912, thenthe proximity sensors of the glove interface object are not utilized toindicate proximity to the device 900.

In the above-described implementations, reference has been made to theproximity or distance of the glove interface object and its proximitysensors to the device 900 and its electromagnets. It is contemplatedthat the determination of the locations of the glove interface objectand the device 900 relative to each other can be performed based on themagnetic tracking methods thus described and/or by other methods, suchas by visual tracking (e.g. capturing images of the interactiveenvironment and employing image recognition to recognize and track theglove interface object and/or the device 900) and the use of motionsensors (e.g. accelerometers, gyroscopes, magnetometers) included in theglove interface object and/or the device 900.

In one embodiment illustrating several of the present concepts appliedin combination, when the glove interface object exceeds the threshold912, then magnetic tracking of the glove interface object relative tothe device 900 is not performed. However, magnetic tracking of thefingertips of the glove interface object relative to itself (e.g. usingelectromagnets on a wrist portion of the glove interface object) may beperformed. The location of the glove interface object in space isdetermined by visual tracking and/or use of motion sensors in the gloveinterface object; and the location of the device 900 in space can alsobe determined by visual tracking and/or use of motion sensors in thedevice 900. When the glove interface object crosses over the threshold912 and moves into the course tracking region (as determined from thevisual tracking and/or motion sensor tracking), then the long-rangeelectromagnets 902 a-d of the device 900 are activated and the proximitysensors of the glove interface object are utilized, at least in part, todetect proximity and/or relative location to the long-rangeelectromagnets. During this time, the activation sequence of thelong-range electromagnets could be multiplexed with the activationsequence of wrist electromagnets of the glove interface object, so thatproximity sensors could indicate proximity/relative location to bothsets of electromagnets during different sampling cycles. When the gloveinterface object crosses over the threshold 910 into the fine trackingregion, then the close-range electromagnets 904 a-c are activated at ahigher activation sequence frequency than that of the long-rangeelectromagnets. The wrist electromagnets of the glove interface objectmay be completely deactivated at this point, so that the proximitysensors indicate proximity to the close-range electromagnetsexclusively.

The specific examples of configurations for utilizing magnetic tracking,including course and fine tracking, in combination with other types oftracking such as visual and motion sensor based tracking, have beendescribed by way of example only and not by way of limitation. Forpurposes of brevity, not all possible combinations of these methods oftracking are described in detail. However, it will be apparent to thoseskilled in the art, that any of the principles and methods discussedherein can be applied in combination with each other to define otherembodiments which are contemplated as part of the present disclosure.

In some implementations, visual tracking is utilized to determine thelocation and/or orientation of a glove interface object in the localenvironment, whereas the magnetic tracking schema described herein isutilized to obtain the fine positioning of the user's hand(s) includingthe positioning of the user's fingers. This information can be utilizedas input for an interactive application.

In another implementation, instead of having two separately definedlong-range and close-range electromagnets, a single set ofelectromagnets may be utilized in a high-power configuration (forlong-range tracking) and a low-power configuration (for close-rangetracking) to achieve similar effects. When the high-power configurationis utilized, the frequency of activation sequences may be reduced ascompared to the low-power configuration, thereby conserving batterypower.

In various implementations, the number of long-range and close-rangeelectromagnets may vary. In different implementations, there may be moreor fewer short-range electromagnets than long-range electromagnets. Inimplementations wherein the same set of electromagnets are utilized forlong-range and close-range tracking, the number of electromagnets whichare activated and utilized may vary in a similar manner.

In some implementations, fine-grained tracking is utilized for rendering(e.g. a corresponding virtual hand), whereas coarse tracking is utilizedfor gesture recognition. In some applications or during certain times ofinteractivity (e.g. virtual hand is not in current view frustum), theremay be no rendering of the corresponding virtual hand, and in suchcircumstances, coarse tracking is sufficient for purposes of gesturerecognition. Whereas when the virtual hand is being actively rendered,fine tracking can be engaged to provide for high fidelity andresponsiveness of the rendering.

Additionally, it will be appreciated that though long-range andclose-range tracking have been described with respect to a peripheraldevice, there may be any number of defined ranges, and/or the samplingrate of the proximity sensors and the frequency of activation sequencesof the electromagnets may be continuously variable according todistance, velocity of movement of the fingers, battery life, as set byan interactive application such as a video game, etc. Furthermore,similar principles may be applied with the glove interface object alone,the sampling rate of the proximity sensors and the frequency ofactivation of the electromagnets of the glove interface object beingvariable according to location of the glove (e.g. in predefinedranges/regions/locations), orientation of the glove, as set by theinteractive application, etc. For example, when a user's hands are in adownward pointing resting position, then the sampling rate and frequencyof activation may be reduced, whereas when the user's hands are raisedfor interactivity, then the sampling rate and frequency of activationmay be increased.

As has been discussed, when a user is wearing an HMD, their view of thelocal environment may be obstructed, and this makes it difficult tolocate and interact with physical devices in the local environment.However, by utilizing a system of emitters on a peripheral device andcorresponding proximity sensors on a glove interface object, theproximity and relative location of the user's hand to the peripheraldevice can be determined. Such information can be utilized to assist auser in finding the peripheral device when they are unable to see it.For example, an indicator could be displayed in the user's view (e.g. ofa virtual environment) that is being rendered on the HMD, wherein theindicator is configured to indicate the location of the peripheraldevice in the local environment. A virtual hand that corresponds to theuser's hand could also be shown, such that the positional relationshipbetween the indicator and the virtual hand is accurately representativeof the relationship between the peripheral device and the user's hand.As such, the user is able to guide their hand to the peripheral deviceon the basis of the displayed scene on the HMD.

In addition to assisting the user in finding a peripheral device whenwearing an HMD, the magnetic tracking methods described herein can beutilized to allow a user to see the positioning of their hands andfingers relative to the peripheral device in the view (e.g. of a virtualspace) that is being presented on the HMD. For example, the virtual handcan be shown interacting with a representation of the peripheral device,wherein the movements of the virtual hand correspond to those of theuser's hand as detected via the glove interface object. In this manner,the user is able to interact with the peripheral device in an intuitivemanner despite not having a direct line of sight, as they are able tosee a corresponding interaction in the virtual space rendered on theHMD.

Additionally, embodiments have been described with reference to ahead-mounted display. However, it should be appreciated that in otherembodiments, non-head mounted displays may be substituted, such as atelevision, projector, LCD display screen, portable device screen (e.g.tablet, smartphone, laptop, etc.) or any other type of display that canbe configured to render video in accordance with the present embodimentsof the invention.

FIG. 10A schematically illustrates a system for interfacing with aninteractive application using a glove interface object, in accordancewith an embodiment of the invention. The glove interface object 1000includes flex sensors 1010, pressure sensors 1012, touch switches 1014,inertial sensors 1016, and biometric sensors 1018. A data streamprocessor 1020 is configured to process data from the various sensors.It will be appreciated that in various embodiments, the data streamprocessor 1020 may process sensor data to various extents, includingdetermining values quantifying sensed activity, identifying poses,gestures, movements, etc. A haptic feedback controller 1022 isconfigured to control the operation of haptic feedback devices 1024. Alight controller 1026 is configured to control the operation of lights1028. A communications interface is configured to communicate datato/from other devices.

The glove interface object further includes an emitter controller 1032that controls the operation of emitters 1034, including the activationand deactivation thereof. Proximity sensor controller 1036 controls theoperation of proximity sensors 1038, including activating (e.g.supplying current to the proximity sensors) and reading the proximitysensors. The emitter controller 1032 and proximity sensor controller1036 can be configured to provide for time-division multiplexing of theactivation/deactivation of the emitters and the reading of the proximitysensors.

A computing device 1040 is configured to execute a video game, andcommunicate with the glove interface object 1000. The video game isrendered on an display/HMD 1042. An image/sound capture device 1044captures images and sound from the interactive environment in which theuser is situated. It should be appreciated that the computing device1040 receives data from the glove interface object such as sensor data,and the computing device may also generate commands to control theoperation of the various devices of the glove interface object 1000, toeffect the functionality of the glove interface object discussed herein.

FIG. 10B illustrates additional components of the computing device 1040,in accordance with an embodiment of the invention. The glove interfaceobject 1000 provides hand gesture data, detected/processed from theglove interface object's various sensors, to a hand gesture identifier1050. The hand gesture identifier 1050 can define a hand pose identifier1052 for identifying a pose of the user's hand, and a hand motionidentifier 1054 for identifying dynamic movements of the user's hand,such as motion and/or changes in the pose of the user's hand. Thesedefine gestures detected from the glove interface object 1000 that aresupplied to a video game 1060 as input. In one embodiment, a gesturelibrary 1056 is provided, containing reference data defining variousgestures, which may be utilized to identify gestures for the video game.

In accordance with embodiments described herein, data indicatingdistances from various sensors to various emitters can be generated andprocessed to determine the relative locations of the sensors to theemitters. This information can be utilized to identify and/or infer thepose of the user's hand. For example, by identifying the location of thefingertips (at which the sensors are disposed) relative to the user'swrist (at which the emitters are disposed), then the user's hand posecan be determined by an inverse kinematic process, includingdetermination of various aspects such as the pose of the user's fingers(including flexion and deviation of the user's fingers, e.g. bend ofspecific joints (e.g. knuckles)) and the pose of the user's wrist(including flexion and deviation of the user's wrist). It will beappreciated that in some embodiments, sensor data can be correlated tohand pose, such by use of a look-up table. In some implementations, amodel of the user's hand is generated by the computing device and inputfrom the glove interface object is utilized to update the model.

Additionally, the game system may control sampling frequencies andemissions from the glove interface object based on the in-game context.

An image data processor 1058 processes images captured by the imagecapture device 1044, to identify trackable objects such as lights on theglove interface object 1000. The hand tracking processor 1062 isconfigured to perform location tracking 1064 and orientation tracking1066 of the hand of the user, based on the identified trackable objectsas well as inertial data 1072 from the glove interface object 1000. Thelocation and orientation of the glove interface object (as defined bythe user's hand) may also be provided as input to the video game 1060.The video game 1060 may generate haptic feedback data 1074 fortransmission to the glove interface object 1000, which thereby producesthe haptic feedback. The video game 1076 may also generate light controldata 1076 for controlling the lights on the glove interface object 1000.Additionally, the video game 1060 generates video/audio data 1078 forrendering by the display/HMD 1042.

In some embodiments, the glove interface object is defined by an innerglove and an outer glove. The inner glove is removable and washable,whereas the outer glove contains the hardware for the glove interfaceobject's functionality as described herein. Additionally, the innerglove may function as an insulator to insulate the hardware of the gloveinterface object from the user.

In some embodiments, haptic feedback can be provided by vibrating thefingertips at various frequencies to simulate textures as a user moveshis fingers along a surface.

In some embodiments, force feedback mechanisms can be included in theglove interface object. Devices can be included which oppose motions ofthe user's hands/fingers, to simulate resistance encountered when makingsuch motions. For example, a force feedback mechanism may oppose themotion of closing one's fingers, thus simulating the feel forgrabbing/holding an object.

In some embodiments, pressure feedback mechanisms can be provided whichapply pressure to at least a portion of the hand as a feedbackmechanism. For example, a clamp may squeeze a finger as feedback, e.g.when touching a virtual object.

It should be appreciated that the input provided by the glove interfaceobject can be applied to provide real-time control of a virtual hand orother object in a virtual environment. In some embodiments, the inputprovided by the glove interface object provides control of anon-hand-like object in the virtual environment, such as enablingmanipulation of the object. In some embodiments, the input provided bythe glove interface object provides real-time control of an arm orhand-like object of a character that is controlled by the user. Whenutilized in the context of presentation on an HMD device, the gloveinterface object can provide a highly immersive and intuitive experiencewith respect to control of an arm/hand or similar appendage of acharacter in the virtual environment. That is, the user can experience asensation as if the virtual arm/hand or appendage really is their ownarm/hand, resulting from the real-time control and responsivenessafforded by the glove interface object in combination with the highlyimmersive presentation of the HMD device.

Furthermore, it will be appreciated that within an interactive sessionof an interactive application, the virtual hand may be shown or notshown depending upon the execution state of the interactive application.For example, in a video game, there may be variousstages/scenes/tasks/levels/etc. that may or may not require the virtualhand to be shown. Furthermore, the rendering of the virtual hand may beshown or not shown in the virtual environment depending upon the contextor content of the virtual environment. For example, the virtual handmight be shown (or made available to be shown) when a specific object ispresent in the virtual scene, or when the user approaches the specificobject to manipulate it or otherwise interact with it.

In some implementations, the pose and/or movement of the user'shand/fingers can define a gesture that can be identified from trackingthe glove interface object in accordance with the principles discussedherein. The identified gesture can be configured to cause some action inthe virtual environment—that is, the gesture is recognized andcorrelated to a produce a specific input for the interactive applicationthat is generating the virtual environment. In various embodiments, avirtual hand may or may not be shown in conjunction with the gestureidentification.

With reference to FIG. 11, a diagram illustrating components of a gloveinterface object 104 is shown, in accordance with an embodiment of theinvention. The glove interface object 104 includes a processor 1100 forexecuting program instructions. A memory 1102 is provided for storagepurposes, and may include both volatile and non-volatile memory. Abattery 1106 is provided as a power source for the glove interfaceobject 104. A motion detection module 1108 may include any of variouskinds of motion sensitive hardware, such as a magnetometer 1110, anaccelerometer 1112, and a gyroscope 1114.

The glove interface object 104 includes speakers 1120 for providingaudio output. Also, a microphone 1122 may be included for capturingaudio from the real environment, including sounds from the ambientenvironment, speech made by the user, etc. The glove interface object104 includes tactile feedback module 1124 for providing tactile feedbackto the user. In one embodiment, the tactile feedback module 1124 iscapable of causing movement and/or vibration of the glove interfaceobject 104 so as to provide tactile feedback to the user.

LEDs 1126 are provided as visual indicators of statuses of the gloveinterface object 104. For example, an LED may indicate battery level,power on, etc. A USB interface 1130 is included as one example of aninterface for enabling connection of peripheral devices, or connectionto other devices, such as other portable devices, computers, etc. Invarious embodiments of the glove interface object 104, any of variouskinds of interfaces may be included to enable greater connectivity ofthe glove interface object 104.

A WiFi module 1132 is included for enabling connection to the Internetor a local area network via wireless networking technologies. Also, theglove interface object 104 includes a Bluetooth module 1134 for enablingwireless connection to other devices. A communications link 1136 mayalso be included for connection to other devices. In one embodiment, thecommunications link 1136 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 1136 mayutilize any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 1138 are included to provide an input interfacefor the user. Any of various kinds of input interfaces may be included,such as buttons, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 1140 may be included in glove interface object 104for facilitating communication with other devices via ultra-sonictechnologies.

Bio-sensors 1142 are included to enable detection of physiological datafrom a user. In one embodiment, the bio-sensors 1142 include one or moredry electrodes for detecting bio-electric signals of the user throughthe user's skin.

The foregoing components of glove interface object 104 have beendescribed as merely exemplary components that may be included in gloveinterface object 104. In various embodiments of the invention, the gloveinterface object 104 may or may not include some of the variousaforementioned components. Embodiments of the glove interface object 104may additionally include other components not presently described, butknown in the art, for purposes of facilitating aspects of the presentinvention as herein described.

It will be appreciated by those skilled in the art that in variousembodiments of the invention, the aforementioned glove interface objectmay be utilized in conjunction with an interactive application displayedon a display to provide various interactive functions. The exemplaryembodiments described herein are provided by way of example only, andnot by way of limitation.

With reference to FIG. 12, a diagram illustrating components of ahead-mounted display 102 is shown, in accordance with an embodiment ofthe invention. The head-mounted display 102 includes a processor 1300for executing program instructions. A memory 1302 is provided forstorage purposes, and may include both volatile and non-volatile memory.A display 1304 is included which provides a visual interface that a usermay view. A battery 1306 is provided as a power source for thehead-mounted display 102. A motion detection module 1308 may include anyof various kinds of motion sensitive hardware, such as a magnetometer1310, an accelerometer 1312, and a gyroscope 1314.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis models are availableto detect magnitude and direction of the acceleration in differentdirections. The accelerometer is used to sense inclination, vibration,and shock. In one embodiment, three accelerometers 1312 are used toprovide the direction of gravity, which gives an absolute reference fortwo angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic fieldin the vicinity of the head-mounted display. In one embodiment, threemagnetometers 1310 are used within the head-mounted display, ensuring anabsolute reference for the world-space yaw angle. In one embodiment, themagnetometer is designed to span the earth magnetic field, which is ±80microtesla. Magnetometers are affected by metal, and provide a yawmeasurement that is monotonic with actual yaw. The magnetic field may bewarped due to metal in the environment, which causes a warp in the yawmeasurement. If necessary, this warp can be calibrated using informationfrom other sensors such as the gyroscope or the camera. In oneembodiment, accelerometer 1312 is used together with magnetometer 1310to obtain the inclination and azimuth of the head-mounted display 102.

In some implementations, the magnetometers of the head-mounted displayare configured so as to be read during times when electromagnets inother nearby devices are inactive.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, threegyroscopes 1314 provide information about movement across the respectiveaxis (x, y and z) based on inertial sensing. The gyroscopes help indetecting fast rotations. However, the gyroscopes can drift overtimewithout the existence of an absolute reference. This requires resettingthe gyroscopes periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 1316 is provided for capturing images and image streams of areal environment. More than one camera may be included in thehead-mounted display 102, including a camera that is rear-facing(directed away from a user when the user is viewing the display of thehead-mounted display 102), and a camera that is front-facing (directedtowards the user when the user is viewing the display of thehead-mounted display 102). Additionally, a depth camera 1318 may beincluded in the head-mounted display 102 for sensing depth informationof objects in a real environment.

The head-mounted display 102 includes speakers 1320 for providing audiooutput. Also, a microphone 1322 may be included for capturing audio fromthe real environment, including sounds from the ambient environment,speech made by the user, etc. The head-mounted display 102 includestactile feedback module 1324 for providing tactile feedback to the user.In one embodiment, the tactile feedback module 1324 is capable ofcausing movement and/or vibration of the head-mounted display 102 so asto provide tactile feedback to the user.

LEDs 1326 are provided as visual indicators of statuses of thehead-mounted display 102. For example, an LED may indicate batterylevel, power on, etc. A card reader 1328 is provided to enable thehead-mounted display 102 to read and write information to and from amemory card. A USB interface 1330 is included as one example of aninterface for enabling connection of peripheral devices, or connectionto other devices, such as other portable devices, computers, etc. Invarious embodiments of the head-mounted display 102, any of variouskinds of interfaces may be included to enable greater connectivity ofthe head-mounted display 102.

A WiFi module 1332 is included for enabling connection to the Internetor a local area network via wireless networking technologies. Also, thehead-mounted display 102 includes a Bluetooth module 1334 for enablingwireless connection to other devices. A communications link 1336 mayalso be included for connection to other devices. In one embodiment, thecommunications link 1336 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 1336 mayutilize any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 1338 are included to provide an input interfacefor the user. Any of various kinds of input interfaces may be included,such as buttons, touchpad, joystick, trackball, etc. An ultra-soniccommunication module 1340 may be included in head-mounted display 102for facilitating communication with other devices via ultra-sonictechnologies.

Bio-sensors 1342 are included to enable detection of physiological datafrom a user. In one embodiment, the bio-sensors 1342 include one or moredry electrodes for detecting bio-electric signals of the user throughthe user's skin.

A video input 1344 is configured to receive a video signal from aprimary processing computer (e.g. main game console) for rendering onthe HMD. In some implementations, the video input is an HDMI input.

The foregoing components of head-mounted display 102 have been describedas merely exemplary components that may be included in head-mounteddisplay 102. In various embodiments of the invention, the head-mounteddisplay 102 may or may not include some of the various aforementionedcomponents. Embodiments of the head-mounted display 102 may additionallyinclude other components not presently described, but known in the art,for purposes of facilitating aspects of the present invention as hereindescribed.

FIG. 13 is a block diagram of a Game System 1400, according to variousembodiments of the invention. Game System 1400 is configured to providea video stream to one or more Clients 1410 via a Network 1415. GameSystem 1400 typically includes a Video Server System 1420 and anoptional game server 1425. Video Server System 1420 is configured toprovide the video stream to the one or more Clients 1410 with a minimalquality of service. For example, Video Server System 1420 may receive agame command that changes the state of or a point of view within a videogame, and provide Clients 1410 with an updated video stream reflectingthis change in state with minimal lag time. The Video Server System 1420may be configured to provide the video stream in a wide variety ofalternative video formats, including formats yet to be defined. Further,the video stream may include video frames configured for presentation toa user at a wide variety of frame rates. Typical frame rates are 30frames per second, 60 frames per second, and 120 frames per second.Although higher or lower frame rates are included in alternativeembodiments of the invention.

Clients 1410, referred to herein individually as 1410A, 1410B, etc., mayinclude head mounted displays, terminals, personal computers, gameconsoles, tablet computers, telephones, set top boxes, kiosks, wirelessdevices, digital pads, stand-alone devices, handheld game playingdevices, and/or the like. Typically, Clients 1410 are configured toreceive encoded video streams, decode the video streams, and present theresulting video to a user, e.g., a player of a game. The processes ofreceiving encoded video streams and/or decoding the video streamstypically includes storing individual video frames in a receive bufferof the client. The video streams may be presented to the user on adisplay integral to Client 1410 or on a separate device such as amonitor or television. Clients 1410 are optionally configured to supportmore than one game player. For example, a game console may be configuredto support two, three, four or more simultaneous players. Each of theseplayers may receive a separate video stream, or a single video streammay include regions of a frame generated specifically for each player,e.g., generated based on each player's point of view. Clients 1410 areoptionally geographically dispersed. The number of clients included inGame System 1400 may vary widely from one or two to thousands, tens ofthousands, or more. As used herein, the term “game player” is used torefer to a person that plays a game and the term “game playing device”is used to refer to a device used to play a game. In some embodiments,the game playing device may refer to a plurality of computing devicesthat cooperate to deliver a game experience to the user. For example, agame console and an HMD may cooperate with the video server system 1420to deliver a game viewed through the HMD. In one embodiment, the gameconsole receives the video stream from the video server system 1420, andthe game console forwards the video stream, or updates to the videostream, to the HMD for rendering.

Clients 1410 are configured to receive video streams via Network 1415.Network 1415 may be any type of communication network including, atelephone network, the Internet, wireless networks, powerline networks,local area networks, wide area networks, private networks, and/or thelike. In typical embodiments, the video streams are communicated viastandard protocols, such as TCP/IP or UDP/IP. Alternatively, the videostreams are communicated via proprietary standards.

A typical example of Clients 1410 is a personal computer comprising aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logic mayinclude hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams arewell known in the art and vary depending on the particular encodingscheme used.

Clients 1410 may, but are not required to, further include systemsconfigured for modifying received video. For example, a client may beconfigured to perform further rendering, to overlay one video image onanother video image, to crop a video image, and/or the like. Forexample, Clients 1410 may be configured to receive various types ofvideo frames, such as I-frames, P-frames and B-frames, and to processthese frames into images for display to a user. In some embodiments, amember of Clients 1410 is configured to perform further rendering,shading, conversion to 3-D, or like operations on the video stream. Amember of Clients 1410 is optionally configured to receive more than oneaudio or video stream. Input devices of Clients 1410 may include, forexample, a one-hand game controller, a two-hand game controller, agesture recognition system, a gaze recognition system, a voicerecognition system, a keyboard, a joystick, a pointing device, a forcefeedback device, a motion and/or location sensing device, a mouse, atouch screen, a neural interface, a camera, input devices yet to bedeveloped, and/or the like.

The video stream (and optionally audio stream) received by Clients 1410is generated and provided by Video Server System 1420. As is describedfurther elsewhere herein, this video stream includes video frames (andthe audio stream includes audio frames). The video frames are configured(e.g., they include pixel information in an appropriate data structure)to contribute meaningfully to the images displayed to the user. As usedherein, the term “video frames” is used to refer to frames includingpredominantly information that is configured to contribute to, e.g. toeffect, the images shown to the user. Most of the teachings herein withregard to “video frames” can also be applied to “audio frames.”

Clients 1410 are typically configured to receive inputs from a user.These inputs may include game commands configured to change the state ofthe video game or otherwise affect game play. The game commands can bereceived using input devices and/or may be automatically generated bycomputing instructions executing on Clients 1410. The received gamecommands are communicated from Clients 1410 via Network 1415 to VideoServer System 1420 and/or Game Server 1425. For example, in someembodiments, the game commands are communicated to Game Server 1425 viaVideo Server System 1420. In some embodiments, separate copies of thegame commands are communicated from Clients 1410 to Game Server 1425 andVideo Server System 1420. The communication of game commands isoptionally dependent on the identity of the command. Game commands areoptionally communicated from Client 1410A through a different route orcommunication channel that that used to provide audio or video streamsto Client 1410A.

Game Server 1425 is optionally operated by a different entity than VideoServer System 1420. For example, Game Server 1425 may be operated by thepublisher of a multiplayer game. In this example, Video Server System1420 is optionally viewed as a client by Game Server 1425 and optionallyconfigured to appear from the point of view of Game Server 1425 to be aprior art client executing a prior art game engine. Communicationbetween Video Server System 1420 and Game Server 1425 optionally occursvia Network 1415. As such, Game Server 1425 can be a prior artmultiplayer game server that sends game state information to multipleclients, one of which is game server system 1420. Video Server System1420 may be configured to communicate with multiple instances of GameServer 1425 at the same time. For example, Video Server System 1420 canbe configured to provide a plurality of different video games todifferent users. Each of these different video games may be supported bya different Game Server 1425 and/or published by different entities. Insome embodiments, several geographically distributed instances of VideoServer System 1420 are configured to provide game video to a pluralityof different users. Each of these instances of Video Server System 1420may be in communication with the same instance of Game Server 1425.Communication between Video Server System 1420 and one or more GameServer 1425 optionally occurs via a dedicated communication channel. Forexample, Video Server System 1420 may be connected to Game Server 1425via a high bandwidth channel that is dedicated to communication betweenthese two systems.

Video Server System 1420 comprises at least a Video Source 1430, an I/ODevice 1445, a Processor 1450, and non-transitory Storage 1455. VideoServer System 1420 may include one computing device or be distributedamong a plurality of computing devices. These computing devices areoptionally connected via a communications system such as a local areanetwork.

Video Source 1430 is configured to provide a video stream, e.g.,streaming video or a series of video frames that form a moving picture.In some embodiments, Video Source 1430 includes a video game engine andrendering logic. The video game engine is configured to receive gamecommands from a player and to maintain a copy of the state of the videogame based on the received commands. This game state includes theposition of objects in a game environment, as well as typically a pointof view. The game state may also include properties, images, colorsand/or textures of objects. The game state is typically maintained basedon game rules, as well as game commands such as move, turn, attack, setfocus to, interact, use, and/or the like. Part of the game engine isoptionally disposed within Game Server 1425. Game Server 1425 maymaintain a copy of the state of the game based on game commands receivedfrom multiple players using geographically disperse clients. In thesecases, the game state is provided by Game Server 1425 to Video Source1430, wherein a copy of the game state is stored and rendering isperformed. Game Server 1425 may receive game commands directly fromClients 1410 via Network 1415, and/or may receive game commands viaVideo Server System 1420.

Video Source 1430 typically includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such asStorage 1455. This rendering logic is configured to create video framesof the video stream based on the game state. All or part of therendering logic is optionally disposed within a graphics processing unit(GPU). Rendering logic typically includes processing stages configuredfor determining the three-dimensional spatial relationships betweenobjects and/or for applying appropriate textures, etc., based on thegame state and viewpoint. The rendering logic produces raw video that isthen usually encoded prior to communication to Clients 1410. Forexample, the raw video may be encoded according to an Adobe Flash®standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith,MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, or the like. Theencoding process produces a video stream that is optionally packaged fordelivery to a decoder on a remote device. The video stream ischaracterized by a frame size and a frame rate. Typical frame sizesinclude 800×600, 1280×720 (e.g., 720p), 1024×768, although any otherframe sizes may be used. The frame rate is the number of video framesper second. A video stream may include different types of video frames.For example, the H.264 standard includes a “P” frame and a “I” frame.I-frames include information to refresh all macro blocks/pixels on adisplay device, while P-frames include information to refresh a subsetthereof. P-frames are typically smaller in data size than are I-frames.As used herein the term “frame size” is meant to refer to a number ofpixels within a frame. The term “frame data size” is used to refer to anumber of bytes required to store the frame.

In alternative embodiments Video Source 1430 includes a video recordingdevice such as a camera. This camera may be used to generate delayed orlive video that can be included in the video stream of a computer game.The resulting video stream, optionally includes both rendered images andimages recorded using a still or video camera. Video Source 1430 mayalso include storage devices configured to store previously recordedvideo to be included in a video stream. Video Source 1430 may alsoinclude motion or positioning sensing devices configured to detectmotion or position of an object, e.g., person, and logic configured todetermine a game state or produce video-based on the detected motionand/or position.

Video Source 1430 is optionally configured to provide overlaysconfigured to be placed on other video. For example, these overlays mayinclude a command interface, log in instructions, messages to a gameplayer, images of other game players, video feeds of other game players(e.g., webcam video). In embodiments of Client 1410A including a touchscreen interface or a gaze detection interface, the overlay may includea virtual keyboard, joystick, touch pad, and/or the like. In one exampleof an overlay a player's voice is overlaid on an audio stream. VideoSource 1430 optionally further includes one or more audio sources.

In embodiments wherein Video Server System 1420 is configured tomaintain the game state based on input from more than one player, eachplayer may have a different point of view comprising a position anddirection of view. Video Source 1430 is optionally configured to providea separate video stream for each player based on their point of view.Further, Video Source 1430 may be configured to provide a differentframe size, frame data size, and/or encoding to each of Client 1410.Video Source 1430 is optionally configured to provide 3-D video.

I/O Device 1445 is configured for Video Server System 1420 to sendand/or receive information such as video, commands, requests forinformation, a game state, gaze information, device motion, devicelocation, user motion, client identities, player identities, gamecommands, security information, audio, and/or the like. I/O Device 1445typically includes communication hardware such as a network card ormodem. I/O Device 1445 is configured to communicate with Game Server1425, Network 1415, and/or Clients 1410.

Processor 1450 is configured to execute logic, e.g. software, includedwithin the various components of Video Server System 1420 discussedherein. For example, Processor 1450 may be programmed with softwareinstructions in order to perform the functions of Video Source 1430,Game Server 1425, and/or a Client Qualifier 1460. Video Server System1420 optionally includes more than one instance of Processor 1450.Processor 1450 may also be programmed with software instructions inorder to execute commands received by Video Server System 1420, or tocoordinate the operation of the various elements of Game System 1400discussed herein. Processor 1450 may include one or more hardwaredevice. Processor 1450 is an electronic processor.

Storage 1455 includes non-transitory analog and/or digital storagedevices. For example, Storage 1455 may include an analog storage deviceconfigured to store video frames. Storage 1455 may include a computerreadable digital storage, e.g. a hard drive, an optical drive, or solidstate storage. Storage 1415 is configured (e.g. by way of an appropriatedata structure or file system) to store video frames, artificial frames,a video stream including both video frames and artificial frames, audioframe, an audio stream, and/or the like. Storage 1455 is optionallydistributed among a plurality of devices. In some embodiments, Storage1455 is configured to store the software components of Video Source 1430discussed elsewhere herein. These components may be stored in a formatready to be provisioned when needed.

Video Server System 1420 optionally further comprises Client Qualifier1460. Client Qualifier 1460 is configured for remotely determining thecapabilities of a client, such as Clients 1410A or 1410B. Thesecapabilities can include both the capabilities of Client 1410A itself aswell as the capabilities of one or more communication channels betweenClient 1410A and Video Server System 1420. For example, Client Qualifier1460 may be configured to test a communication channel through Network1415.

Client Qualifier 1460 can determine (e.g., discover) the capabilities ofClient 1410A manually or automatically. Manual determination includescommunicating with a user of Client 1410A and asking the user to providecapabilities. For example, in some embodiments, Client Qualifier 1460 isconfigured to display images, text, and/or the like within a browser ofClient 1410A. In one embodiment, Client 1410A is an HMD that includes abrowser. In another embodiment, client 1410A is a game console having abrowser, which may be displayed on the HMD. The displayed objectsrequest that the user enter information such as operating system,processor, video decoder type, type of network connection, displayresolution, etc. of Client 1410A. The information entered by the user iscommunicated back to Client Qualifier 1460.

Automatic determination may occur, for example, by execution of an agenton Client 1410A and/or by sending test video to Client 1410A. The agentmay comprise computing instructions, such as java script, embedded in aweb page or installed as an add-on. The agent is optionally provided byClient Qualifier 1460. In various embodiments, the agent can find outprocessing power of Client 1410A, decoding and display capabilities ofClient 1410A, lag time reliability and bandwidth of communicationchannels between Client 1410A and Video Server System 1420, a displaytype of Client 1410A, firewalls present on Client 1410A, hardware ofClient 1410A, software executing on Client 1410A, registry entrieswithin Client 1410A, and/or the like.

Client Qualifier 1460 includes hardware, firmware, and/or softwarestored on a computer readable medium. Client Qualifier 1460 isoptionally disposed on a computing device separate from one or moreother elements of Video Server System 1420. For example, in someembodiments, Client Qualifier 1460 is configured to determine thecharacteristics of communication channels between Clients 1410 and morethan one instance of Video Server System 1420. In these embodiments theinformation discovered by Client Qualifier can be used to determinewhich instance of Video Server System 1420 is best suited for deliveryof streaming video to one of Clients 1410.

Embodiments of the present invention may be practiced with variouscomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theinvention can also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a wire-based or wireless network.

With the above embodiments in mind, it should be understood that theinvention can employ various computer-implemented operations involvingdata stored in computer systems. These operations are those requiringphysical manipulation of physical quantities. Any of the operationsdescribed herein that form part of the invention are useful machineoperations. The invention also relates to a device or an apparatus forperforming these operations. The apparatus can be specially constructedfor the required purpose, or the apparatus can be a general-purposecomputer selectively activated or configured by a computer programstored in the computer. In particular, various general-purpose machinescan be used with computer programs written in accordance with theteachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data, which can be thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications can be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the present disclosure.

What is claimed is:
 1. A method, comprising: using an image capturedevice to capture images of an interactive environment; processing thecaptured images to track a location of a trackable object on a gloveinterface object in the interactive environment, the tracked location ofthe trackable object defining a location of a virtual hand in a virtualenvironment; receiving finger proximity data from a plurality ofproximity sensors positioned at fingertip portions of the gloveinterface object, the finger proximity data indicating distances to eachof a plurality of emitters positioned on the glove interface object;processing the proximity data to identify locations of the fingertipportions relative to the emitters; applying the locations of thefingertip portions to define poses of virtual fingers of the virtualhand in the virtual environment for rendering on a head-mounted display.2. The method of claim 1, wherein receiving finger proximity data isdefined from activation of the emitters and reading of the proximitysensors in a time-division multiplexed arrangement, wherein each of theproximity sensors is read during activation of a single emitter, whereinthe time-division multiplexed arrangement is defined by a repeatedpattern of activation of the emitters that provides for activation ofeach of the emitters during separate time periods, and reading of eachof the proximity sensors during the time period of activation of a givenone of the emitters.
 3. The method of claim 1, wherein the plurality ofemitters defines at least three electromagnets that are positioned on awrist portion of the glove interface object in a non-collineararrangement.
 4. The method of claim 3, wherein the plurality ofproximity sensors defines five magnetic sensors respectively positionedat five fingertip portions of the glove interface object.
 5. The methodof claim 1, wherein the trackable object is illuminated to facilitateidentification in the captured images.
 6. The method of claim 1, furthercomprising: receiving orientation data from an orientation sensordefined on the glove interface object; processing the orientation datato define an orientation of the virtual hand in the virtual environment.7. The method of claim 6, wherein the orientation sensor is selectedfrom the group consisting of an accelerometer, a gyroscope, and amagnetometer.
 8. A method, comprising: activating a plurality of gloveemitters positioned on a glove interface object; using a plurality ofproximity sensors positioned at fingertip portions of the gloveinterface object to determine a proximity of the fingertip portions tothe glove emitters; in response to determining a location of the gloveinterface object within a predefined distance of a peripheral device,activating a plurality of peripheral emitters positioned at theperipheral device, and transitioning, from using the proximity sensorsto determine the proximity of the fingertip portions to the gloveemitters, to using the proximity sensors to determine a proximity of thefingertip portions to the peripheral emitters.
 9. The method of claim 8,wherein transitioning includes terminating the activation of the gloveemitters.
 10. The method of claim 8, wherein activating the gloveemitters defines a cyclical activation sequence of the glove emitters,wherein during a single activation time period for a given gloveemitter, proximity data is read from each of the proximity sensors. 11.The method of claim 8, wherein determining the location of the gloveinterface object within the predefined distance of the peripheral deviceincludes processing captured image data of an interactive environment toidentify the location of the glove interface object and a location ofthe peripheral device.
 12. The method of claim 8, wherein determiningthe location of the glove interface object includes activating asecondary peripheral emitter on the peripheral device, and using theproximity sensors to determine a proximity of the fingertip portions tothe secondary peripheral emitter; and wherein transitioning includesterminating the activation of the secondary peripheral emitter.
 13. Themethod of claim 8, wherein the glove emitters are positioned at a wristportion of the glove interface object.
 14. The method of claim 8,wherein the glove emitters and the peripheral emitters are defined byelectromagnets, and the proximity sensors are defined by magneticsensors.
 15. A system for interfacing with an interactive application,comprising: a glove interface object, the glove interface objectincluding, a plurality of glove emitters, a plurality of proximitysensors positioned at fingertip portions of the glove interface object,the proximity sensors being configured to indicate a proximity of thefingertip portions to the glove emitters, a glove controller configuredto control activation of the plurality of glove emitters and reading ofthe proximity sensors; and, a peripheral device, the peripheral deviceincluding, a plurality of peripheral emitters, a peripheral controllerconfigured to activate the peripheral emitters in response to a locationof the glove interface object being determined within a predefineddistance of the peripheral device; wherein the proximity sensors,further in response to the location of the glove interface object beingdetermined within the predefined distance, transition, from indicatingthe proximity of the fingertip portions to the glove emitters, toindicating a proximity of the fingertip portions to the peripheralemitters.
 16. The system of claim 15, wherein the glove controller isconfigured to terminate the activation of the glove emitters in responseto the location of the glove interface object being determined withinthe predefined distance.
 17. The system of claim 15, wherein the glovecontroller is configured to define a cyclical activation sequence of theglove emitters, wherein during a single activation time period for agiven glove emitter, proximity data is read from each of the proximitysensors.
 18. The system of claim 15, wherein the peripheral devicefurther includes, a secondary peripheral emitter, wherein the peripheralcontroller is configured to control the activation of the secondaryperipheral emitter, the peripheral emitter being used for determiningthe location of the glove interface object; wherein the glove controlleris configured to read the proximity sensors to determine a proximity ofthe fingertip portions to the secondary peripheral emitter that definesthe location of the glove interface object; and wherein the peripheralcontroller is configured, further in response to the location of theglove interface object being determined within the predefined distance,to terminate activation of the secondary peripheral emitter.
 19. Thesystem of claim 15, wherein the glove emitters are positioned at a wristportion of the glove interface object.
 20. The system of claim 15,wherein the glove emitters and the peripheral emitters are defined byelectromagnets, and the proximity sensors are defined by magneticsensors.